Logistic Regression

Hi. I've just written some code for logistic regression. My data is a
two-author discrimination problem, with one author labelled as 0 and
the other as 1. The x values are the frequences of a list of common
words in training texts, ten per author. So, it's easy to have more
dimensions (I did include an extra column of 1s for the intercept) than
I have rows of data.

I used Newton-Rhapson to learn the weights. The method I used is
similar to that on the page:

http://www.google.co.uk/url?sa=U&start=2&q=http://ocw.mit.edu/NR/rdonlyres/Sloan-School-of-Management/15-075Applied-StatisticsSpring2003/8C07CE0F-70BB-4C8F-9A7B-9AD0AF643D71/0/lec15_logistic_regression.pdf&e=10313

I start with a 0 vector for the initial set of weights.

(I can't find the actual web page I copied the weight update formula
from).

When I apply my program to test problems, it usually quickly finds a
very close fit, where my 10 "0" texts are assigned a probability very
close to zero, and my 10 "1" texts are assigned a probability very
close to one. However, sometimes, particularly when I have 10 or more
dimensions, I either find a fit that assigns a probability of 0 to all
texts, or simply fails to converge (sum of absolute values of
differences to be added to the weights < 0.01) in any reasonable time.
Usually convergence happens in a very short time.

My data should be simple. In fact, the first variable alone (frequency
of 'the') is sufficient to distinguish the two authors I'm using for
debugging.

Are the problems I'm seeing likely to be due to the search technique
I'm using, or is it more likely that I have a bug in my program, and
N-R should always converge to a good solution?

Cheers,

Ross-c

.

Relevant Pages

  • Re: Logistic Regression
    ... where my 10 "0" texts are assigned a probability very ... I either find a fit that assigns a probability of 0 to all ... >differences to be added to the weights < 0.01) in any reasonable time. ...
    (sci.stat.math)
  • Re: BAYESIAN (weighted) PROBABILITY ??
    ... > where Pis the prior probability. ... liklihood ratio actually already contains the weights. ... that GOD=G_uv is CORRECT, given the scientific evidence". ...
    (sci.stat.math)
  • Re: ISO help in probability
    ... Helmut Giese wrote: ... > probability by multiplying a random value with the associated weight, ... intervals based on the weights you have. ... proc create_wtable {var weights} { ...
    (comp.lang.tcl)
  • Re: Combining Probabilities?
    ... I have two different predictors which generate a probability between ... One predictor has a limited set of symbols it will generate ... You can also use sets of weights selected by some small ... In paq1 I counted 0 and 1 bits in each context, ...
    (comp.compression)
  • Re: How do I sample randomly based on some probability(wightage)?
    ... I need to randomly sample from a list where all choices have weights ... In short I mean if prob of a H is .9 and probability of T be 0.1 then ... Replace each item with the sum of all the items up to and including itself. ...
    (comp.lang.python)