Re: Questions about square errors

On May 2, 1:15 pm, "aggie2525" <aggie2...@xxxxxxxxxxx> wrote:
I was not the one who defined 9 input parameters.
Based upon experts in the field that I study, these 9 input parameters are
important to determine and control the outcome of the output.
However, my concern is that these 9 input parameters may not absolutely
independent to each other.

I actually use neural networks to find the best fit solution.
I divided my data into 3 different groups (one is for training, another is
for validating. The last one is for testing).
These 3 different data groups are used to make sure that I do not over-train
and create over-fitting solutions best suiting only
data that I used in training.

Thus, each iteration (which include the whole run on training data sets), I
should have the mean square error.
I will record only the neural network that produces the least mean square
error.
I feel that the least mean square error may not sufficient to tell us how
well my program perform.
I am thinking about using the correlation between the predicted and observed
outputs to be control my neural network training also.
Am I correct on this one?

In stead of one predicted output for each data set (item) feed in the model,
I expect to be able to provide a range of outputs with
a certain confidential level as defined by users.

In my mind, if I can normalize this error (residual) distribution curve, I
may be able to give users a range of outputs when users provide a
confidential level that they want.

Another concern is that I am also afraid that the error (residual)
distribution curve may be biased toward certain input patterns.
Thus, the distribution curve of residuals may not be a good indicator to
provide an error of each predicted outcome.

Do you have any comment on this?
How should I handle this problem?

Thank you very much.

"Old Mac User" <chendrixst...@xxxxxxxxx> wrote in messagenews:1178127812.620791.191160@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

On May 2, 9:38 am, "aggie2525" <aggie2...@xxxxxxxxxxx> wrote:
Hi,

I am working on a model that use about 9 input parameters to predict an
output.
Since I have about 800 data sets (each data set has 9 input data and an
output).

I come up with a method that I can predict an output from 9 given input
data.
Then, I use the model that I have to predict each output for each set of
9
input data.
As a result, I have a square error for each prediction.
Therefore, there should be about 800 square errors

My question is if it is OK that I plot all 800 square errors to get their
distribution.
Then, from this distribution curve, I can get a range of errors with a
specified confidential level for my prediction.

My concern is that the square errors would be dependent to certain input
patterns.

Thank you in advance for any help and reply.

If you have minimized the sum of squares of the differences between
observed and predicted values, then it appears you may have invented
least squares and/or multiple regression.

It's smart to examine and study the differences between observed and
predicted values. In multiple regression, those are called residuals.

But there's a lot more to it than this. For instance, it's a good bet
that your nine predictors are to some degree correlated among
themselves. Your model may have several unnecessary predictors. If so,
then the presence of those probably degrades the capability of your
model for predicting future outcomes. The fact that it may predict
existing data fairly well is not necessarily an indicator of how well
it will predict future experiences.

Then there's the matter of the significance (or absence of
significance) of each of the individual predictors and the confidence
intervals on the estimated value of each predictor.

All of these should be taken into account before moving forward.

If you are not familiar with... and skilled in... the analysis of
multivariable data, then I suggest you get some help with this
project. OMU

I forgot to mentiokn something. You said...

" These 3 different data groups are used to make sure that I do not
over-train
and create over-fitting solutions best suiting only
data that I used in training."

Therein lies the reason for calling NNs "multiple regression without
ethics". There's actually no way to determine how many fitting
constants you have in the NN model.

We've run experiments in fitting data with NNs. In these experiments
we generated "data" by using random numbers coming from a shuffled
deck of cards combined with rolling dice. Given those "data", the NN
software fitted those random numbers delightfully. We had no clue to
how many fitting constants (neurons!!) were used to fit those random
numbers. When you come to the bottom line, there's no way to tell
when NN software is overfitting because that kind of software does not
produce valid statistical measures as guidance. OMU

.

Relevant Pages

  • Re: Questions about square errors
    ... I was not the one who defined 9 input parameters. ... I feel that the least mean square error may not sufficient to tell us how ... In my mind, if I can normalize this error distribution curve, I ... Your model may have several unnecessary predictors. ...
    (sci.stat.math)
  • Re: Questions about square errors
    ... I feel that the least mean square error may not sufficient to tell us how ... Your model may have several unnecessary predictors. ... Overfitting: The use of more weights than necessary to adequately ... Algorithms that automatically delete intermediate processing nodes ...
    (sci.stat.math)
  • Re: Questions about square errors
    ... I feel that the least mean square error may not sufficient to tell us how ... In my mind, if I can normalize this error distribution curve, I ... In multiple regression, those are called residuals. ... Your model may have several unnecessary predictors. ...
    (sci.stat.math)
  • Comparing predictors
    ... to measure the quality of Y' by the mean square error E^2). ... I now have the suspicion that the neural network predictor Y'' = g(X_1, ... So far the only way sensible way I can think of for comparing predictors ... in my case the correlation ...
    (sci.stat.math)
  • Re: Multicollinearity !!!!!
    ... I interpret multicollinearity as saying that the predictors have some information in common, and each has some unique information to contribute. ... Assuming there's no external information that will let you determine which predictors are most expendable, you pretty much have to pick one, drop it and take your chances. ... One thing I suggest to my students is to identify which predictors are collinear, then run the model with each dropped individually and look at the residuals. ...
    (sci.stat.math)