A/B Testing on mobile Hamburger menu vs menu
I am doing a A/B testing on our mobile site to see hambergar menu is beporming better or the Menu with the boarder. The test has been running for the past 2 weeks and the unique visitorts are 76,125 and the statistical significane is less than 1%. I am wondering how long more does the test needs to run to reach the 90%. I am aassuming to get the 90% the test needs to run for couple of months, do I need to wait that long or can I make a decision and pick the winner.
Solved! Go to Solution.
For a good test you probably need more than 95% statistical significance. What's probably happening in this case is that the numbers for both variants are too close together and that you can't draw a clear winner from the two of them. Usually what we'd call that is an inconclusive winner as you're not able to call even a winner or a loser. You're probably off testing something else then running this for a couple of months as you'll have to many fluctuations usually in that period to make sure it's a real winner or loser.
Hope this answers your question!
Have another question, when it comes to e-commerce total revenue is more important or success?