Rich-text Reply

Deciding traffic distribution

rmilyard 10-26-18
Accepted Solution

Deciding traffic distribution

Hi all!


So my team and I are discussing when it is best to do accerlated impact vs manual. There seem to be differing opinions on what is best: using manual to obtain statistical significance and splitting distribution 50/50 versus running an experiment on accerlate impact, not worrying abotu statistical significance and just obtaining conversions.





Level 2

JasonDahlin 10-26-18

Re: Deciding traffic distribution

Making decisions to "just obtain conversions" based on small sample sizes likely lead to making poor choices, meaning you actually have fewer conversions because you are chasing early returns where false-positives make the lesser option appear greater.

If you want to know what is best long-term, then go with 50-50 so you have better chance of not implementing bad changes into your baseline.

If your issue is time-sensitive (like testing a blog post headline that is only going to generate traffic for a few hours or a few days), then you're probably not going to get statistical significance anyway (unless one variation really really sucks), go with the Bandit approach.
--Jason Dahlin
Analytics and Testing Guru Smiley Happy

Experimentation Hero