Rich-text Reply

Statistics: Power in relation to type of test, statistical assumptions

Jeroen 10-27-14
Accepted Solution

Statistics: Power in relation to type of test, statistical assumptions

Hi fellow testers,

 

 

What is the approach to find out the amount of visitors for MVT test, or an ABCD test?

In articles, like the ones shown below, I find information about significance and how to deal with power for an A/B test, but not for other types of tests.

 

So, eg. could I use the 'A/B test sample size calculator' in the context of an MVT or ABCD test? (just use the outcome for each cell, variant?)

 

 

Kind regards,

Jeroen

 

 

 

See articles:

Interpreting More Complex Results

 

and

 

How many visitors do I need for A/B test?

 

 

Level 2

adzeds 10-27-14
 

Re: Statistics: Power in relation to type of test, statistical assumptions

Hi Jeroen,

 

A tool that might be able to help you out is:

 

http://www.houseofkaizen.com/conversion-rate-optimisation/resources/calculators/test-length-estimate...

 

If you fill in each section and hit go it will tell you approx how many days you will need to test for based on a rough estimate of the traffic that will hit the page you are testing.

 

Hope that helps

 

 

David Shaw
Level 11
Jeroen 10-27-14
 

Re: Statistics: Power in relation to type of test, statistical assumptions

thx David
Level 2
Hudson 10-31-14
 

Re: Statistics: Power in relation to type of test, statistical assumptions

[ Edited ]

Hi Jeroen, 

 

You can use Optimizely's Sample Size Calculator to understand how to calculate the sample size of each variation.

 

The logic is the same for an A/B test as for MVTs and A/B/n tests; you'll have more variations and therefore more total traffic, but the calculation is the same per variation. 

 

Happy testing!

 

Hudson

 

 

 

 

 

 

Optimizely