Re: Nonlinear Least-Squares curve-fitting


Old Mac User wrote:
Cafei...

If you are finding approximately the same residual sum of squares for
various combinations of the fitting parameters, then you have very poor
estimates of those parameters. In other words, the confidence intervals
on the parameters are very wide.

It always goes back to the "design of the experiments" that spawned the
data. Attempting to fit a "complex" nonlinear model to just any old set
of accumulated data is a gross waste of time. Estimating parameters in
a linear model can be a loser game if the "independent variables" are
poorly arranged in their space. Attempting to do this with a "complex"
nonlinear model is many times worse. Worse in the sense of wasting time
and effort.

Multiple minima... or what seem to be multiple minima... many of which
have approx. the same residual sum of squares... suggests your data is
poorly conditioned for the model you are attempting to fit.

You may be right. Let me describe what I did in more detail, so that
perhaps you can explain why this happened:

The data that I got came from physics experiments. The functions to fit
the data came from standard physics equations. The parameters in these
equations were actually real physics quantities which could be
experimentally measured. When I applied the algorithm which I described
of gradually adding data points, the parameters that the algorithm
output were much closer to the experimentally observed values for the
parameters than when I did not do it gradually.

However, when I did the same thing except that I chose the parameters a
priori and I artificially generated data to fit the parameters for
which the error in the data was normally distributed, the algorithm
which I described of gradually adding data points did not give any
improvement over when I did not do it gradually.

Any possible explanations?


So go back and look at those residual sums of squares. At the end of
the "fitting" process, are many of them approx. the same? OMU


cafeinst@xxxxxxx wrote:
Scott Seidman wrote:
cafeinst@xxxxxxx wrote in news:1158947866.574595.138520
@h48g2000cwc.googlegroups.com:

This technique was suggested to me in 1992 when I was having trouble
getting my computer program to give a nonlinear least-squares fit to a
certain complicated function. And it worked amazingly. I'd like to know
if anyone has heard of this?

Craig



Never heard of this, but I would think that whether it would work nicely or
not would depend upon how well-behaved the function and the data are--
sometimes it might work nicely, and sometimes not-so-nicely.

I have also tried it on other types of functions, but there didn't seem
to be any gain in doing it this way. The intuition for this type of
technique is that if you start with a small number of data points, the
least squares function is not as complicated as if you had started with
a large number of data points; therefore, if the least squares function
involves a small number of data points, there won't be as many local
minima to avoid as if the least squares function involved a large
number of data points.

It's using *gradualness* to solve the problem. It's actually quite
natural and probably explains why it is not good to teach 1st graders
how to read by giving them "Huck Finn". It's better to first teach them
"Hellicopters and Gingerbread" or "A Duck is a Duck" and let them read
"Huck Finn" in high school. This concept can be used for Machine
Learning too, I would think.

Craig


Regardless of the method you use to generate your initial guess, best
practice is to start from a variety of locations in n-space, and see which
initial guess gives you the smallest least squares error. You usually
don't want to depend on just one initial guess to give you the "right"
answer, even if you have a nifty algorithm like the one you described.
Choosing that handful of starting points can be something of an art, too.

There are some algorithms that actually provide the estimator with enough
"energy" to jump out of local minima. You see this type of thing often
with neural nets, where there can be a hundreds of (essentially
meaningless) weights to estimate, and there are local minima all over the
place.

--
Scott
Reverse name to reply

.

Relevant Pages

  • Re: Not complicated, weird situation
    ... of squares in some way that involves factoring a number you get to ... This is a faaaaantastic algorithm. ... > better than 50% success rate, which is a claim you made but didn't ... > hard work, Harris' work broke into society with the help of Justin, ...
    (sci.crypt)
  • Re: A very fast Fermat factoring algorithm
    ... there is a way to speed up Fermat's method by considering only the squares ... there are a several known techniques to speed Fermat ... > I'd rather not share the technique just yet. ... > nothing like Fermat's algorithm, but it has the same runtime ...
    (sci.crypt)
  • Re: Mutual Visibility Field Of View
    ... I agree that if you send out enough rays for a given radius, ... But the bottleneck is in visiting all 10,000 squares (my map ... of the reasons why I developed my algorithm. ... of previous walls that have been encountered in a view. ...
    (rec.games.roguelike.development)
  • Re: Math/Programming Algorithm Help
    ... I'm having trouble with an algorithm. ... grid of size NxN, they need to click to activate N number of squares, so ... that all the activated squares are contiguous, ie, they each squares shares ... You might also read a little bit about graph theory. ...
    (comp.lang.java.help)
  • Avoiding random trial and error
    ... every time random trial and error" algorithm. ... computing time and memory than a "deterministic" algorithm would take. ... no matter how rare were the valid squares and how many of them ... method did it, while the "shuffle" errored. ...
    (rec.games.roguelike.development)
Loading