How does the current stats engine handle different % traffic allocation for variations?
How does the stats engine handle different % traffic allocation for variations?
For example, a conservative stakeholder wants the original gets 75% and the variation 25%.
Solved! Go to Solution.
It will just take longer to reach significance. Everything is a ratio and the significance is constrained on # of success events and conversion rate, so if you go from 50% to 25% of traffic on one variation, that means you're sending half the traffic to that variation and you'll take twice as long to get to significance than you would otherwise. To get a sense of the magnitude of that change, use the sample size calculator (https://www.optimizely.com/resources/sample-size-calculator/) and the sample size will be the traffic needed for the 25% part of it. IOW, double the traffic needed from the sample size calculator to get the sample size needed for your 25/75 split.
One other tactic that has worked for me to appease the internal stakeholder is to start at 25/75 and make sure the results aren't doing something crazy, and then when it's looking like a win, move to 50/50 to get to significance that it's actually a win, and not just noise in the data.