Re: comparing prediction accuracy between two MLR models

On 5 Sep 2006 03:09:05 -0700, "Benoît" <benoit.depaire.reg@xxxxxxxxx>
wrote:

Hello,

Is there a statistical test to compare the prediction accuracy between
two MLR models.

the models are nested, if that's of any relevance.


What does your MLR stand for? Multiple linear regression?

Nested models are usually testable by using the "improvement
in R^2" divided by the d.f.-difference as the numerator, taking
the residual (1 minus R^2) divided by its d.f. as the denominator.
Every book on regression should have this. Sometimes called the
Chow test.


Furthermore, both models produce a classification table. i've found a

Classification tables are an incidental byproduct of the fit.
You don't ordinarily test them.

book that mentions a binomial d statistic to see if the predictive
accuracy is significantly different from prediction by chance (always
predicting the mode of the dependent variable). however, the book also
mentions one should use the binomial test for a difference of two
proportions (Bulmer, 1979, p.145) if one would like to compare the
predictive accuracies between two models.

Does anybody has any idea or some good references?

The are definitely "correlated proportions" so the simplest
test on differences won't be right. I have no idea what Bulmer
gives. What I found, right off, Googling groups on "Correlated
proportions" -

David Winsemius posted these references a month ago --

"Exact confidence limits for the difference between two correlated
proportions" Chris J. Lloyd and Max V. Moldovan
<http://www.mbs.edu/download.cfm?DownloadFile=951E3FAB-123F-A0D8-429F1...>

"A New Confidence Interval for the Difference Between Two
BinomialProportions of Paired Data" Xiao-Hua Zhou Gengsheng Qin
<http://www.bepress.com/cgi/viewcontent.cgi?article=1028&context=uwbio...>

David also posted some code, so you might want to read his post -
http://groups.google.com/group/sci.stat.consult/msg/562597a82376d05b?hl=en&;

--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Loading