Re: Logistic Regression

In article <1117188427.541938.211140@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>,
clemenr@xxxxxxxxxx writes

[...]
>Now, I think can see why the weights would go to infinity if the
>problem is too easy, as the instances are labelled 1 and 0 and,
>assuming infinite numerical precision, then likelihood will always
>increase the further the probabilities of the instances are pushed
>towards extreme values, and reaching 1 and 0 requires infinite weights,
>no? In the case where the classes cannot be perfectly separated, the
>decreasing increase in likelihood obtained by pushing the probabilities
>of the correctly classified instances towards the extremes will
>eventually be outweighed by even a few misclassified instances, no?

I'll let Ray explain his table, but you've got the basic point now. The
way I'd put it is that linearly separable means there a vector a and
scalar b so that a.x+b is always negative when x comes from one class
and positive when x comes from the other. (a and b define the separating
hyperplane.) Your probabilities are of form (1+exp(a.x+b))^{-1} so if
such a,b exist, you can make the probabilities closer to 0 or 1 by
multiplying a and b by a number bigger than 1.

>I do get good fits sometimes for more than 9 attributes. However, my
>program does give up when the change in weights is less than a
>constant. As the derivative of the logistic curve gets closer and
>closer to 0 as the probability nears 1 or 0, I presume that the changes
>in the weights will also slow. Hence, my program may cease iterations,
>and return a set of weights that haven't reached maximum likelihood.

I think the change in weights should be fast with unlimited accuracy,
but with limited accuracy there will come a point where
(1+exp(a.x+b))^{-1} is indistinguishable from 0 or 1.

>I
>also presume that where my program goes crazy and starts assigning
>silly probabilities when it was "almost there" is due to floating point
>overflow in the matrix algebra. If that is the case, then it might be
>possible to prevent this by putting a strict limit on the absolute
>values of weights. This of course would mean that I'm not going to get
>the maximum likelihood fit, but that may be better than no fit at all.

Yes. You don't really want the ML fit anyway. There are various ways of
dealing with the situation, and I'm not sure which is most suitable for
you. A prior on the weights, or jitter or weight decay as used in neural
nets, and no doubt others.

--
Graham Jones
http://www.visiv.co.uk
Emails to graham@xxxxxxxxxxx may be deleted as spam
Please add a j just before the @ to ensure delivery

.

Relevant Pages

  • Re: Integration weights and points
    ... > more information/convergance properties. ... The integral is infinite if ... I am looking for only weights and points as I am trying to solve ... using matrix methods ...
    (sci.math.num-analysis)
  • Re: The Runners Physique
    ... rare exceptions) is able to do it without being already fit. ... This is like saying you need to lift heavy weights to be muscular. ... lifting alone, but you can build muscles at the expense of fat loss. ... Depending how much off your target weight you are, lifting alone can be ...
    (misc.fitness.weights)
  • Re: Fitting: unbalanced data point density and weights
    ... When I plot them in a log-log scale they ... coefficients of the linear combination are the parameters of the fit). ... smaller (I use the errors to define weights, ... that the regression function is not really a power function, ...
    (sci.stat.math)
  • Re: Fitting reference curves to experimental data. Help developing an algroithm
    ... I have some experimental data and several reference data. ... So far i am able to fit a polynomial to the data sets and I'm trying ... each spectra were collected. ... provided that you know the weights wj ahead of time. ...
    (sci.math)
  • Re: Software suggestions
    ... > a "good" set of weights. ... > If this is the case, then you are in trouble. ... > learning is far from a solved problem. ... returns the equation that gives the best fit. ...
    (sci.stat.math)