Rich-text Reply

What to Do When Your A/B Test Doesn’t Win

richpage75 05-10-16

What to Do When Your A/B Test Doesn’t Win

[ Edited ]

Hi all,


Did you know that 1 in 7 A/B tests won't get a winning result? 


Rather than throwing away your losing tests and hoping for better luck next time, you have a huge learning and research opportunity to improve your future A/B test ideas and get much better results.


To help ensure you succeed with your A/B tests and maximize your learnings from results, I’ve put together a handy checklist. Hopefully you will find it very useful. 


I would love to hear your thoughts and whether you have tried any of methods I mention. 





Get my free conversion rate optimization toolbox:
robertchan 05-24-16

Re: What to Do When Your A/B Test Doesn’t Win

When an experiment doesn't win, I archive it into a historical repo so that it can be used in the future for reference in the ideation of another experiment. I also go over the results with relevant heads and departments to see what may have contributed to the results.
Robert Chan

Experimentation Hero
MartijnSch 05-25-16

Re: What to Do When Your A/B Test Doesn’t Win

Well if 6 out of 7 tests get you a winner that is not the original I would be very surprised because that's highly unlikely ;-). But for the tests that we do and don't reach significance we usually test them longer. We have very short, 7 day cycles and then tend to run them a bit longer.

Just for some elements you'll never reach enough conversions in a set timeframe to have the right significance. We learned along the way that we then just don't test it at all.