Rich-text Reply

How does the current stats engine handle different % traffic allocation for variations?

pcward 09-14-15
Accepted Solution

How does the current stats engine handle different % traffic allocation for variations?

Hello,

How does the stats engine handle different % traffic allocation for variations?

 

For example, a conservative stakeholder wants the original gets 75% and the variation 25%.  

 

Thanks!

Level 2

CROmetrics 09-14-15
 

Re: How does the current stats engine handle different % traffic allocation for variations?

It will just take longer to reach significance.  Everything is a ratio and the significance is constrained on # of success events and conversion rate, so if you go from 50% to 25% of traffic on one variation, that means you're sending half the traffic to that variation and you'll take twice as long to get to significance than you would otherwise.  To get a sense of the magnitude of that change, use  the sample size calculator (https://www.optimizely.com/resources/sample-size-calculator/) and the sample size will be the traffic needed for the 25% part of it.  IOW, double the traffic needed from the sample size calculator to get the sample size needed for your 25/75 split.

 

One other tactic that has worked for me to appease the internal stakeholder is to start at 25/75 and make sure the results aren't doing something crazy, and then when it's looking like a win, move to 50/50 to get to significance that it's actually a win, and not just noise in the data.