Re: Significance of mean square error decrease between different models


erik_arner@xxxxxxxxxxx wrote:
Hi,

Let's say I have some data that I want to try to fit into different
models.

First I try linear regression, with a resulting mean square error.

Then, using a non-linear model based on some reasonable assumptions
about the phenomenon I'm modelling, I come up with a "better"
approximation, i.e. a lower mean square error than the linear model.

How do I determine if the lower error of the second model is
statistically significant? What assumptions about the data do I need to
make?

Thanks,
Erik

Erik,

We at AFS run into this all the time .. as we test for model
improvements OVER different types of modesl.

To begin with you must compare apples to apples ...not for example a
model without a transformation in Y versus a model e.g. in the
reciprocal of Y . The sum of squares or MSE have to be in COMPARABLE
TERMS ..thus one would have to compute the FITTED VALUES in the
original metric and then compute the MSE and perform a classic F Test
to see if there is a statistically significant difference.

The problem becomes a little more complex when you have ARIMA structure
or lagged values in either Y or X which yield FEWER fitted values thus
the MSE's would not be directly comparable as both MSE's were computed
for a different number of points.

If you want to discuss this further , I would be glad to take your call
...

Glad to help ..

Dave Reilly
Automatic Forecasting Systems
http://www.autobox.com
215-675-0652

.

Relevant Pages

  • Re: Significance of mean square error decrease between different models
    ... First I try linear regression, with a resulting mean square error. ... using an "information criterion" such as the Akaike Information ... this amounts to using an objective function that is ...
    (sci.stat.math)
  • Significance of mean square error decrease between different models
    ... Let's say I have some data that I want to try to fit into different ... First I try linear regression, with a resulting mean square error. ... How do I determine if the lower error of the second model is ...
    (sci.stat.math)
  • Re: Regressing against a proportion?
    ... organ. ... They want to compare the effect between treatment groups; in particular, they would like to see a 20% or better difference. ... If they are not too close to either end of the scale, linear regression will work quite well. ...
    (sci.stat.edu)