Re: Sample Size for Online Pay-Per-Click Advertising

From: Richard Ulrich (Rich.Ulrich_at_comcast.net)
Date: 12/27/04 

Date: Sun, 26 Dec 2004 20:10:00 -0500

On 26 Dec 2004 13:58:42 -0800, "espeed" <james@unifiedmind.com> wrote:

> I am trying to determine how long we should test an online
> pay-per-click advertising campaign before we can have a good idea of
> how it will perform if we let it run continuously.
>
> I need to find the sample size, but I am not sure if it can be
> considered a random sample since we will not be selecting the subjects
> at random, they will be selecting us by choosing to click on the ad.
> Further, since market conditions change over time, a sample that
> represents the population now may not represent the population a few
> weeks later so the test may be useless for projections. So, am I
> testing a sample or am I only testing the entire population for a
> shorter period of time?
>
> I am not a statistics expert so please let me know if my analysis is
> correct...

You are certainly naming a number of the critical
reasons why a simple power analysis can't be used
to project an N -- no random sample, resampling the
same people an unknown number of times, change of the
underlying *conditions* in just a few weeks so that
extrapolation is difficult in any degree.

Given those difficulties, and others, I think that you have
a very specialized area where experience and insight
(gained partly by debriefing users) is going to be more
important than statistical evidence.

>
> An Overture ad for a given keyword costs $10.65 per click for the top
> position. I hypothesize that we will maximize profits by paying for it
> and letting it run continuously.
>
> We can test this theory if we can commit to paying for enough clicks to

I don't see how that leads to a test of the theory.
If you have an alternate way of "letting it run",
you could compare two ways for specified times,
but that doesn't necessarily give a *good* test,
partly because of the reasons you stated -- for
instance, circumstances changing quickly.

> equal the sample size of the total possible clicks for a given
> click-through rate (CTR), for a given time period. Overture provides
> estimates on how many impressions we will have for the month so once we
> determine the CTR for our ad, we can estimate the total number of
> clicks we would have if we let the ad run for an entire month.
>
> The total number of clicks for the month is the population size (notice
> that the population size is not the number of people whom see the ad,
> but it's the number of people whom click on the ad because we are
> testing the performance of the Web page).
>
> For example, let's assume a 2% click-through rate and 612,740 ad
> impressions for the month so at a 2% CTR, that would result in 12,254.8
> clicks for the month. If you set the population size to 12,254.8 with a
> confidence level of 95% and a confidence interval of 2, the sample size
> would be 2,008 clicks at a total cost of $21,385.20.
>
> If the conversion rate for the sample is 4.93%, 2,008 clicks will
> result in 98.9944 conversions. Because we are using a confidence
> interval of 2, we can be 95% sure that if market conditions do not
> change and we let the ads run for the entire month, for the entire
> relevant population, we would have a conversion rate between 2.93%
> (4.93%-2) and 6.93% (4.93%+2).

It seems to me that you will have a better estimate of
the range for the conversion rate if you can collect
data for every day. That provides (a) a little control
for people who are hitting the site three times in the
same day (for instance), and (b) information that might
be even more important, about any declining trend in
clicks across time. - I guess I think that is important,
because it seems to me to be another obvious
"alternate hypothesis", so I could be all wrong about
this one.

>
> Please poke holes in my analysis and let me know if I making any false
> assumptions, any leaps of logic, etc? Or, please let me know how to
> change the test to make it more valid.

Hope this helps. 

-- 
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html