Re: Statistical Analyses of Non-Static Group Question

Hi Callahan,

OK, a few more questions if that's OK, and then hopefully I can start
being useful... :)

1. First, one question I should have thought to ask right off: What's the
typical sample size and range? That is, for each month's summary, how many
active participants are there, how many inactive ones, and how have those
numbers varied over time? This might become fairly important in terms of
deciding what kinds of analyses are possible or feasible.

2. Second, have you been doing any statistical analyses on this data
already? I presume not, but if you have a direction you've been heading
already, I should know that.

3. Third, based on what you've written so far, I couldn't identify any
within- or between-participant differences or manipulations. Put
differently, it seems like there's just one group of participants, and that
they are all experiencing the same thing and responding to the same
measures. Is this correct?

I think that's it for questions. It sounds like what you have in place is
working well for you -- up to the limits of the approach, and it seems
you've done a fine job identifying those limits. I'm not sure that
"statistics" per se is will be the fix to those limits, but it may might
play some role.

In considering all this, the idea of "usable or useful bundles of
information" comes to mind. It's as if you have this stream of information
coming in from each participant. A participant's information isn't terribly
useful until enough and certain kind of it has accrued. Conversely, there
can come a point in time where the information continuing to come in from a
participant starts to become less useful, because it's no longer adding
*incremental* benefit relative to what the participant has already
contribute (and perhaps at times just the act of reporting activity is being
conflated with more meaningful contribution?).

So, I'm wondering if the real "fix" here is to work on clarifying what it
means for the group as a whole to be performing well. It seems as though
there might be two (or more) things at play here that, once conceived of
more distinctly, might open the doors to figuring out how to best proceed.
Specifically, I'm wondering if more clearly separating "reporting
requirements" for being in the product testing program from successfully
making a "unit of contribution" to the program might be helpful. This is
not to say that one is completely excluded from leading up to the other, but
perhaps only up to a certain extent. In other words, I would think that
participants reporting their usage (or lack of usage) of the product is in
and of itself important, and if they do just that, they are making a
contribution to the overall effort, and thus ought earn at least a few
"contribution points" each reporting cycle just for that. To earn
contribution points beyond this basic level, though, they will need to have
contributed additional meaningful information and, in particular, not be
just repeating what they've been saying before. Of course, this might be
very contingent on whether they encountered any problems with the product --
something probably not under their control. So, I would imagine too that
it's important not to penalize or score poorly a participant just because he
or she was "unlucky" enough not to have had any problems with the product.
:)

I could keep going, but I'll stop here -- no need to keep elaborating if I'm
not on the right track. Is what I'm saying in these last 2-3 paragraphs
making sense and resonating in any way? If so, great, and we can keep
moving forward from there. If not, then, if you can, let me know what's
"off" about the direction my thinking is now headed.

The more I consider all this, the more I'm starting to think that statistics
isn't so much the answer as is determining the right way to conceive of and
organize the data. As odd or simple as it may sound, statistics -- for the
most part -- is really just a tool for knowing how and when it's OK to say
that two numbers are different from each other, and that these differences
are "true" and not due to chance or some measurement error. Best I can
tell, these don't seem to be the kind of issues blocking your way forward.

Hope this helps, and I look forward to hearing your response.

Cheers!
Doug


On Thu, 09 Aug 2007 21:01:39 -0700, gwcallahan1@xxxxxxxxxxxx
<gwcallahan1@xxxxxxxxxxxx> wrote:

The participants of program provide product testing feedback in
various forms such as problem reports, status reports, surveys, and
forum discussions. Each from of feedback is weighted with respect to
its importance to the program. On a periodic basis (monthly) each
participant's feedback contribution is counted and weight adjusted to
produce a score. A mean is calculated based only on those participants
that contributed and a rating is awarded based on the participants
score relative to the mean. Non-contributors automatically receive the
lowest possible rating for the period.

The participants for the program are largely employees of the company
that produces the product; only a few outside participants are allowed
for security reasons. Many of the participants are intimate with
particular aspects of the product, however very few have complete
knowledge. The vast majority are simply users of the product with a
layman's understanding of how the product works.

The purpose for measuring participation is manifold and all closely
interrelated: 1) to ascertain that the product is actually being used
and tested, 2) to weed out those who are simply taking advantage of
the benefits of the program without complying with the requirements,
3) to manage resource allocation (placing the product in the hands of
those who will comply with the program requirements versus those who
won't), and 4) to provide some form of feedback to the participants in
hopes of encouraging those who are seriously under-performing to do a
better job.

While it's fairly simple to understand how well a particular
participant is performing on a periodic basis, it's not so easy to
determine how well the group is performing due to the dynamic makeup
of the group (as described in my previous post). I am hoping there's a
way to make statistical adjustments so that I can say with some amount
of certainty whether the performance of the group is better or worse
and by how much.

I hope provides better background on the problem.
.

Relevant Pages

  • Re: Commercial Alarm - help
    ... Well, dispite circular views of perception, some American taxpayers actually ... "They are required to do this for crime statistics in their area." ... >Do you think that they are reporting the truth. ... >> Where do you suppose the Alarm Industry receives crime statistics. ...
    (alt.security.alarms)
  • Re: Commercial Alarm - help
    ... And why do they report these statistics to the USDOJ? ... Do you think that they are reporting the truth. ... > Where do you suppose the Alarm Industry receives crime statistics. ... >> when it is in fact not a false alarm. ...
    (alt.security.alarms)
  • Re: theres a disappointing surprise
    ... the UCR is only for purposes of reporting national ... It uses 8 index crimes: ... > definitions of rape for purposes of charges and trial. ... the statistics from other forms of assault? ...
    (soc.motss)
  • Re: 2.6.13-rc5 randconfig kernel build errors
    ... I'll settle:) It is this feedback I need at ... without reporting stuff to the list. ... I looked at that AmlCode error source file and grepping for a match ... send the line "unsubscribe linux-kernel" in ...
    (Linux-Kernel)
  • Re: Non-Payers
    ... >> in retaliation for reporting that one and now it appears I have three ... >> more non payers. ... > I don't think retaliation is a given. ... that if I reported him he would ruin my perfect feedback rating. ...
    (alt.marketing.online.ebay)