Rich-text Reply

Calculating sample size for an experiment

siiis 06-07-16

Calculating sample size for an experiment

I have two images that I A/B test with one having a CTR=2,4% and I run second image only 1.5% of the time. How to determine how long I should run experiment to see if there is any difference in CTR of images? 

Level 2

siiis 06-07-16
 

Re: Calculating sample size for an experiment

I mean how mathematically derive the right time, not using one of the online calculators.
Level 2
CouchPsycho 06-07-16
 

Re: Calculating sample size for an experiment

What is the right time? This is the key in your question, there is nothing like a right time. You may calculate a number of users, maybe, but most calculators do work with an uplift specified, which makes it somehow a guess.

 

What is the case behind your question? Do you want to schedule a test? Do you want to know, when to stop it if the is no final result?

 

Within an idea of an uplift, there is no calculation possible. Additionally a significance level is necessary. Than you may use chi-square formula to calculate the sample size. Combining sample size with visitor frequency on your page and variation distribution leads then to a time frame. But keep in mind, that only unique visitors are relevant.

 

To keep a long story short: Have a view on the bias in order to determine when to stop an experiment.




"the essence of the creative act is to see the familiar as strange." (anonymous)
siiis 06-07-16
 

Re: Calculating sample size for an experiment

[ Edited ]

Thank you for the response, @CouchPsycho. I have looked at several calculators for sample size (for example, this one) and I wonder what is the logic behind it. I know that the time is not the right word for it, probably the number of users to show image B to know that the results are statistically valid. For significance level I assume is to be 95% and MDE=12% relative. So, my question is how to calculate a sample size without calculator? I do it to understand really what's going on and what can the trade offs.

Level 2
CouchPsycho 06-07-16
 

Re: Calculating sample size for an experiment

If you are familiar with mathematics and statistics you might want to read this:

https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test

If you transform the formula you will make your way thru :-)



"the essence of the creative act is to see the familiar as strange." (anonymous)
siiis 06-07-16
 

Re: Calculating sample size for an experiment

Seems complicated. Which formula should I transform?
Level 2
CouchPsycho 06-08-16
 

Re: Calculating sample size for an experiment

Are you sure, you want to do that. It is higher mathematics.... and if you consider this chi-square to be complicated you probably are going to lose yourself in theory instead of advancing in testing.

May i give you an advice? Do not try to calculate necessary visitors/users. Go ahead and develop tests, learn and get a feeling of how long a test may last depending on the section of your website. That is what i do. A test needs its time....



"the essence of the creative act is to see the familiar as strange." (anonymous)
robertchan 06-10-16
 

Re: Calculating sample size for an experiment

If you're not using an online calculator then you're assumption of when you'll hit statistical significance is as hypothetical as your experiment. You cannot correlate the percentages you specified because your experiment will be dependent on changes that don't necessarily contain historical data. Mathematically, you can imagine the value you're attempting to solve for is 'does not exist' if you're attempting to think of the situation theoretically.
Robert Chan

TanyaMoore 06-23-16
 

Re: Calculating sample size for an experiment

Hi siis,

The great thing about using the Optimizely Calculator is that it does the maths for you as well as having the power of Optimizely's Stats Engine. But if you want to understand the mathematical theory here is a link: https://julienlenestour.com/maths-behind-minimum-sample-size-ab-testing/

Here is also a link that goes into Optimizely's Stats Engine: https://www.optimizely.com/statistics/

Hope thats what you were looking for!

Tanya
siiis 06-23-16
 

Re: Calculating sample size for an experiment

Thanks for the link Tanya. I found it interesting.

I found another calculator that gives a chance to specify what the proportion between A and B. Also there is a reference to a book, which explains how to get answer for arbitrary inputs.
http://clincalc.com/stats/samplesize.aspx
Level 2