Re: ecommerce Conversion Rate Analysis

Steve wrote:

> I have an ecommerce website selling things on the internet.
> I have historical daily data on numbers of users and orders
> (and so hence conversion rate). My problem is that I have
> recently made a change to my site that I think has increased
> conversion, but not by such a large degree that the effect
> is obvious. Average daily conversion has gone up
> slightly since the change, but this could also be due to chance.
>
> Is there a statistical test I can use to give a probability
> to whether I have made a beneficial change or not?

Conventional statistical tests establish something
called "statistical significance" which is different
from practical significance, and which has the
annoying property that as the sample size increases,
a smaller and smaller change can still be
statistically significant.

E-commerce? You're probably working with pretty
large sample sizes. You can forget about conventional
tests; almost any microscopic difference will be
statistically significant.

My advice is just consider the difference in
practical terms. Say the increase in conversion is 5%.
Is a 5% increase enough to repay the cost of the
website development? How much revenue does a 5%
increase represent, and how does that compare to
your operating costs?

In any event, whether 5% is statistically significant
is, at best, a diversion.

A book that might be helpful is Hastie, Tibshirani,
and Friedman, "The Elements of Statistical Learning".
Here's a web page: http://www-stat.stanford.edu/~tibs/ElemStatLearn/

For what it's worth,
Robert Dodier

.