Re: Power Analysis and Multiple Regression and Indifference

On 11 Oct 2006 11:19:04 -0700, "Ben-Bard" <benhoen@xxxxxxxxxxxxx>
wrote:

Bob,

I appreciate your comments. I still need to get at the crux of the
problem...which is sample size. I believe there must be a trade off in
non-psycologocal studies between sample size and effect size. There
would be little reason to visit 1,000,000 homes to garner a result that
is statistically significant yet so close to zero as to be indifferent
from zero.

The literature is fairly extensive in the area of hedonic analysis of
the contribution environmental features have on home values, yet, given
a statistically non-significant result (p-value higher than 0.05) most
conclude there is "no evidence an effect exists". Or conversely if an
effect exists, and is statistically significant, there is little effort
to address if the effect is inside of an amount that would be
characterized as zero.

We have already determined that an effect inside of +/- 3% is close
enough to zero to be ingnored. Therefore if our model returns a result
which has 95% confidence intervals inside of +/- 3% we feel we have
done our job of ruling out if an effect exists. (I suspect we can be
criticized on this front, but we very well might be criticised for
spending thousands of dollars to find a result that is, for most
homeowers, no different from zero.)

The analagous literature has used between 200 and 3000 home sales to do
what we are attempting to do. For us the difference between 200 and
3000 is about $15,000. That's money we would rather not spend. Is
there not some way to establish an appropriate sample size?

Let me review the problem as stated, and point out what is
missing.

You can sample from houses with and without a bad Feature,
where 15-20% have the Feature. The feature is "interesting" if
the 95% CI for the difference is greater than 3%. You are going
to compare two groups on a continuous variable.

Ben-Bard posted before --
"other info: Sample mean $102,000, SD $52,000, Covaried
Standard Dev $25,000."

As Reef Fish posted --
The large SD (and skew), along with calling the difference "3%"
instead of $3000, suggests that the numbers might be considered
after log-transform, instead of as they are.

The regressed/ covaried SD is enough smaller to reduce the needed
sample size by 4-fold; OMU has warned that there could be a lack
of independence between the Feature and those covariates. This
could be a serious drawback; but, otherwise, the "covaried SD" is
the basis for converting the effect size to SD units, instead of the
raw SD.

MISSING.
What is still missing is the "real, underlying difference" that has
to be assumed for a statement of statistical power.
- If the "real difference" is exactly 3%, you have no more than
a 5% chance of excluding it from the 95% CI, no matter how
large the N is. RIght?
- If the "real, underlying difference" is 40%, it won't take a large
N to confirm that there is a 3%+ difference. Right?

The short way to compute the needed N is probably start with the
power analysis in a usual way for detecting a difference from *0*.
For that, you can find out that, for a
- two-tailed t-test at 5%,
- for the power of 80%,
- when the underlying difference is (say) $5000 ( = 0.2 SD),
--> you need an N of about 400 per group.
You should be able to get that result from any power program.
(And other Ns for other assumed Powers and Differences.)

And then, since the difference that matters is not zero but is $3000,
add the $3000 to the statement of the "underlying difference"
($5000, above), so it would take an assumed $8000-difference to
give 80% power of seeing a $3000 difference.



--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Relevant Pages

  • Re: Power Analysis and Multiple Regression and Indifference
    ... is statistically significant yet so close to zero as to be indifferent ... You can sample from houses with and without a bad Feature, ... power analysis in a usual way for detecting a difference from *0*. ... some socioiogist who calls Type 2 error a probability, ...
    (sci.stat.math)
  • Re: The Case is Altered
    ... daisy chaining a monitor's power supply. ... by Symantec Ghost cloning the original disc to a new disc. ... a swivel feature. ... Windows Explorer won't let you do. ...
    (uk.local.surrey)
  • The Case is Altered
    ... daisy chaining a monitor’s power supply. ... by Symantec Ghost cloning the original disc to a new disc. ... a swivel feature. ... Windows NT, see if you can cadge copies of files WINFILE.EXE, ...
    (uk.local.surrey)
  • Re: Question about "Another look at reflections" article.
    ... Correction, it must be moving past a point, not just moving laterally ... average power anywhere on a wire containing a pure standing wave. ... there is zero net energy flow anywhere on a pure standing ...
    (rec.radio.amateur.antenna)
  • Re: OT: Questions about Set Theory
    ... "The number of items with the feature of being over 1000 in the subset as ... "Your current answer is matching mine: zero items, the empty set, ... If you had the ability to comprehend in a normal manner Snit you would have ...
    (comp.sys.mac.advocacy)