Re: linear least squares problems with weights using SVD

From: Allen McIntosh (nospam_at_mouse-potato.com)
Date: 01/27/05 

Date: Thu, 27 Jan 2005 09:18:24 -0500

mikeschuster@gmail.com wrote:

> For solving linear least squares problems I know three methods, two bad
> ones and a good one.
> a) by LU decomposition of the normal equations
> -> bad when occuring matrix (nearly) singular
> b) by Cholesky decomposition of the normal equations
> -> bad when occuring matrix (nearly) non-positive-definite
> c) by SVD
> -> good, takes longer but doesn't fail
You missed d) QR decomposition of the X matrix. It's not quite as good
as SVD in ill-conditioned problems, but it seems to be the method of
choice for "normal" problems these days.
>
> How do I solve linear least squares problems with different weights for
> each data vector by a modification of method c) such that it can handle
> weights?
There is lots of code out there for doing weighted QR decompositions.
If your problem is large, ook for one that works row-wise. Gentleman's
algorithm works row-wise, allows weights, and the original paper has
comments about dealing with singularities. No doubt there are newer
ones available.

Since weighted least squares amounts to doing ordinary least squares on
the matrix sqrt(W)*X where W is the diagonal weight matrix, you could
approach using a simple SVD that way.