Re: scale equivariance

On Feb 4, 9:12 am, "David Jones" <dajx...@xxxxxxxxx> wrote:
elodie wrote:
On Feb 3, 7:59 am, "David Jones" <dajx...@xxxxxxxxx> wrote:
elodie wrote:
On Feb 2, 4:13 pm, elodie <elodie.gill...@xxxxxxxxx> wrote:
Hi everyone

I would like to know why scale equivariance is a desirable property
of the estimator of the coefficients of a linear regression. I
understand that we would like the estimates to follow a change of
scale of the predictors, I was wondering if there was other
reasons.

Thanks in advance

Also I would be interested in knowing whether the Lasso estimator of
the coefficients of a linear regression is scale equivariant. The
Lasso estimator is obtained by minimizing of an objective function,
not by a closed form equation. So I am guessing that the Lasso
estimator is not scale equivariant, but I do not have a solid
argument to support what I think.

Thanks again.

(i) It is not just a change in scale, the usual regression estimates
are invariant to changes of both location and scale of the
observations. The same applies to the predictions created using the
regession estimates.  

(ii) For estimates defined via minimising an objective function, you
will get invariance if the objective function is defined in a way
that allows invarince to happen ... so is theobjective function
simply multiplied by a scale factor if a scale factor is applied to
the observations when an assumption is made that the estimates
behave in this way ...one component may behave as a square while
another is linear in the scale. It may be possible to redefine
penalty function coefficients so as to make the final estimates
invariant.        

David Jones

Thanks a lot for the messages.

The objective function of the Lasso is
Sum(ObsY-Intercept -slope*ObsX)^2+tuning_parameter*sum(abs(Intercept
+Slope)) //equation1
And the minimization is done with respect of the intercept and slope.

Now if we replace ObsX with scalecoeff*ObsX, it looks like the
objective function is different, not just a scaled version of equation
1. So I would conclude that the intercept and slopes will not be
invariant.

Your expression for the objective function seems to be wrong. There may be several different versions of "Lasso" around ... The only paper I found touching on this made the assumption that there was an intial step in which the regressors were standardised so that the design matrix X satisfies X(transp)X=I. If this inital stepo is done then any overall procedure will be invariant to changes of scale of the original regressor variables.

David Jones

Thanks a lot for the message. Thanks a lot for the note on the
standardization of the design matrix and the effect on equivariance.

For the record, the objective function I have for the Lasso is
Sum(ObsY-Intercept -slope*ObsX)^2+tuning_parameter*{abs(Intercept)+abs
(Slope)} //equation1
.

Relevant Pages

  • Re: scale equivariance
    ... the coefficients of a linear regression is scale equivariant. ... Lasso estimator is obtained by minimizing of an objective function, ...
    (sci.stat.math)
  • Re: Error in svd within fmincon
    ... How large are the values of the objective function? ... One way to combat this is to scale your problem. ... There is a failsafe option to check if your objective or ... Hi Steve, thanks for your answer. ...
    (comp.soft-sys.matlab)
  • Re: Error in svd within fmincon
    ... How large are the values of the objective function? ... One way to combat this is to scale your problem. ... There is a failsafe option to check if your objective or ... Hi Steve, thanks for your answer. ...
    (comp.soft-sys.matlab)
  • Re: Error in svd within fmincon
    ... How large are the values of the objective function? ... One way to combat this is to scale your problem. ... There is a failsafe option to check if your objective or ... Hi Steve, thanks for your answer. ...
    (comp.soft-sys.matlab)
  • Re: Error in svd within fmincon
    ... How large are the values of the objective function? ... One way to combat this is to scale your problem. ... There is a failsafe option to check if your objective or ... Hi Steve, thanks for your answer. ...
    (comp.soft-sys.matlab)