Re: Calculating Standard Deviation

On 2006-03-21 16:56:32 -0400, Jentje Goslinga <goslinga@xxxxxxxxx> said:



Robert Israel wrote:
In article <1142890259.473783.308590@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>,
<sunburned.surveyor@xxxxxxxxx> wrote:

Here is the first of my statistics/least squares questions...


When calculating the standard deviation for a set of values you use the
sum of all the residuals squared.


Is this done to remove the tendency for residuals that result from
random errors to sum to zero?


If this is correct, why is the sum of the absolute value of the
residuals not used?


Because you want the standard deviation, not the mean absolute deviation.


I should not that the square root of the sum of the residuals squared
is not the same as the sum of the absolute value of the residuals.


Yes, you should note that.


(So
the two different methods result in different values for the standard
deviation.)


Not "different values", different things entirely.


If there is another reason why the sum of the residuals squared is used
when computing the standard deviation, instead of the sum of the
absolute value of the residuals?


Squares have much nicer properties than absolute values. For example,
there's a nice formula for the variance of a sum, but not for the mean absolute deviation of a sum. You can solve least-squares problems
using linear algebra, but the corresponding mean-absolute-deviation problems require linear programming.

However, I think the OP has a point here: mathematicians
like to reach out for the L2 norm (the sum of squares),
since it has nicer properties, where the user may have
valid reasons to minimize absolute values.
Consider the common problem of trying to fit a line to
some cloud of points in the plane with the requirement of
minimizing the absolute distances, the projections of the
points upon to the line.
Note that there are two points of difference here: (1)
the use of distance rather than ordinate and (2) the use
of absolute values.
I think that the use of absolute values does not lead to
a LP problem: at each iteration of the non-linear
approximation problem when the parameters of the line are
modified you check all the points to see if their position
with respect to the line (same side as the origin) has
changed and flip the sign of the projection as required.
Maybe that is some kind of a LP problem, I am not sure.

Robert Israel israel@xxxxxxxxxxx
Department of Mathematics http://www.math.ubc.ca/~israel University of British Columbia Vancouver, BC, Canada

Just my simple thoughts,

Jen

If you are projecting onto the line rather than using the distance
up to the line then you have wondered over into the land of "errors
in variables" and/or "orthogonal distance regression". Google is
your friend so ask it about both. Sometimes this gets called total
sum of squares so you can try that as well.

Most of these objections have occurred to many folks many times in
the past. There are many too many to list them in an introductory
course/book. So they only show up in advanced books/courses and often
have the difficulty that they require "industrial grade" (i.e. graduate
school level) mathematics to discuss with any level of completeness
or correctness. So yes it is true that the introductory material is
often lacking in its ability to deal with many of the rude facts about
the real world. (That is why you need to staid awake in math classes! ;-))


.

Relevant Pages

  • Re: Calculating Standard Deviation
    ... sum of all the residuals squared. ... Because you want the standard deviation, not the mean absolute deviation. ...
    (sci.math.num-analysis)
  • Re: Best formula
    ... prediction. ... The result is called the "sum of squared errors" or the "sum of absolute ... for under "least squares". ...
    (sci.math)
  • Re: Best formula
    ... prediction. ... The most common measures involve taking all the differences between predicted values and actual values and either squaring them or taking their absolute values and then adding them up. ... The result is called the "sum of squared errors" or the "sum of absolute errors" as appropriate, ... These processes are often collectively called the method of "least squares", and can be searched for under "least squares". ...
    (sci.math)
  • Re: Why is it "least squares" and not "least absolute diffrerence"?
    ...  The least squares method however is quite sensitive to outliers in the ...  for "robust" regression the sum of absolute deviations is used, ...  problem results in a large constraint optimization problem which to solve ...
    (sci.math.num-analysis)
  • Re: Calculating Standard Deviation
    ... sum of all the residuals squared. ... Because you want the standard deviation, not the mean absolute deviation. ...
    (sci.math.num-analysis)