Calculating sample size for an experiment
I have two images that I A/B test with one having a CTR=2,4% and I run second image only 1.5% of the time. How to determine how long I should run experiment to see if there is any difference in CTR of images?
What is the right time? This is the key in your question, there is nothing like a right time. You may calculate a number of users, maybe, but most calculators do work with an uplift specified, which makes it somehow a guess.
What is the case behind your question? Do you want to schedule a test? Do you want to know, when to stop it if the is no final result?
Within an idea of an uplift, there is no calculation possible. Additionally a significance level is necessary. Than you may use chi-square formula to calculate the sample size. Combining sample size with visitor frequency on your page and variation distribution leads then to a time frame. But keep in mind, that only unique visitors are relevant.
To keep a long story short: Have a view on the bias in order to determine when to stop an experiment.
Thank you for the response, @CouchPsycho. I have looked at several calculators for sample size (for example, this one) and I wonder what is the logic behind it. I know that the time is not the right word for it, probably the number of users to show image B to know that the results are statistically valid. For significance level I assume is to be 95% and MDE=12% relative. So, my question is how to calculate a sample size without calculator? I do it to understand really what's going on and what can the trade offs.
If you transform the formula you will make your way thru :-)
May i give you an advice? Do not try to calculate necessary visitors/users. Go ahead and develop tests, learn and get a feeling of how long a test may last depending on the section of your website. That is what i do. A test needs its time....
The great thing about using the Optimizely Calculator is that it does the maths for you as well as having the power of Optimizely's Stats Engine. But if you want to understand the mathematical theory here is a link: https://julienlenestour.com/maths-behind-minimum-sample-size-ab-testing/
Here is also a link that goes into Optimizely's Stats Engine: https://www.optimizely.com/statistics/
Hope thats what you were looking for!
I found another calculator that gives a chance to specify what the proportion between A and B. Also there is a reference to a book, which explains how to get answer for arbitrary inputs.