Rich-text Reply

Tests with very low Baseline Conversion Rate

jerrys 11-27-14

Tests with very low Baseline Conversion Rate

I'm planning some tests on Ads and I'm having some issues with the sample size.


My company don't want to compromise large amouts of traffic for our tests. However I need to test 2 variations where the difference is the position of ads on a given page. 


The main goal is CTR, how many people click on Ads. I want to be able to measure the changes on the number of clicks on ads depending on the ads position.


However the CTR on ads are usually very low (around 0.15%) and for that reason I'm setting the Baseline Conversion Rate at 0.15%.


However in that case the sample data that I need in order to have accurate results is enourmous. Around 212.000 per variation.


Any suggestions about how can I perform this test with a smaller sample size. I believe that I can't get an approval of a sample larger than 70.000.


I imagine that people running tests on low traffic websites who are testing clicks on ads might end up having problems to run this kind of tests or maybe having to run the tests for a long time.



Level 2

adzeds 11-27-14

Re: Tests with very low Baseline Conversion Rate

Hi @jerrys

Sample size is often a problem when looking to run a test such as this.

If you cannot get a large sample set then you might have to use some experience to decide whether or not the test is delivering a successful outcome.

How different is the placement of the ads on the page? If the change is only a subtle one then you may find it difficult to reach an outcome, however, if it is a big difference that leads to a larger gap in performance then you may be able to get away with a smaller sample size a lot easier.

At the end of the day it is up to you to make a call on your test results, so if you are happy with the results you see after 70k visits then go for it!
David Shaw
Level 11
jerrys 11-27-14

Re: Tests with very low Baseline Conversion Rate

Thank you @adzeds 


That's a test for our adaptive site. I'll change the ads position from the top to the content and see if there's a change in performance.


I was wondering if it was a way around. It looks like it doesn't 


I'll try to work with a smaller sample and see if I can get results. 





Level 2