Rich-text Reply

A/B Testing on mobile Hamburger menu vs menu

rarnona 04-04-16
Accepted Solution

A/B Testing on mobile Hamburger menu vs menu

Hi,

 

I am doing a A/B testing on our mobile site to see hambergar menu is beporming better or the Menu with the boarder. The test has been running for the past 2 weeks and the unique visitorts are 76,125 and the statistical significane is less than 1%. I am wondering how long more does the test needs to run to reach the 90%. I am aassuming to get the 90% the test needs to run for couple of months, do I need to wait that long or can I make a decision and pick the winner.

 

Thanks,

Ramona

Level 1

MartijnSch 04-05-16
 

Re: A/B Testing on mobile Hamburger menu vs menu

Hi Ramona,

For a good test you probably need more than 95% statistical significance. What's probably happening in this case is that the numbers for both variants are too close together and that you can't draw a clear winner from the two of them. Usually what we'd call that is an inconclusive winner as you're not able to call even a winner or a loser. You're probably off testing something else then running this for a couple of months as you'll have to many fluctuations usually in that period to make sure it's a real winner or loser.

Hope this answers your question!
rarnona 04-05-16
 

Re: A/B Testing on mobile Hamburger menu vs menu

Thanks Martin, then I let it run to get to 95% before making the decision.
Have another question, when it comes to e-commerce total revenue is more important or success?

Thanks,
Ramona
Level 1
MartijnSch 04-05-16
 

Re: A/B Testing on mobile Hamburger menu vs menu

No please don't focus just on total revenue. Conversion rate is usually a far better metric as it's focusing mostly on the conversions in total. What otherwise could happen is that your metrics might be flawed based on the fact that certain products in a certain variant have a higher margin or price. You'd like to make sure that this won't have an impact on your test and that's why usually you would look at CR%.