Rich-text Reply

What to Do When Your A/B Test Doesn’t Win

richpage75 05-10-16

What to Do When Your A/B Test Doesn’t Win

[ Edited ]

Hi all,

 

Did you know that 1 in 7 A/B tests won't get a winning result? 

 

Rather than throwing away your losing tests and hoping for better luck next time, you have a huge learning and research opportunity to improve your future A/B test ideas and get much better results.

 

To help ensure you succeed with your A/B tests and maximize your learnings from results, I’ve put together a handy checklist. Hopefully you will find it very useful.
http://rich-page.com/cro/ab-testing-results/ 

 

I would love to hear your thoughts and whether you have tried any of methods I mention. 

 

Cheers,

 

Rich

Get my free conversion rate optimization toolbox:
http://services.rich-page.com/toolbox-signup/
robertchan 05-24-16
 

Re: What to Do When Your A/B Test Doesn’t Win

When an experiment doesn't win, I archive it into a historical repo so that it can be used in the future for reference in the ideation of another experiment. I also go over the results with relevant heads and departments to see what may have contributed to the results.
Robert Chan

MartijnSch 05-25-16
 

Re: What to Do When Your A/B Test Doesn’t Win

Well if 6 out of 7 tests get you a winner that is not the original I would be very surprised because that's highly unlikely ;-). But for the tests that we do and don't reach significance we usually test them longer. We have very short, 7 day cycles and then tend to run them a bit longer.

Just for some elements you'll never reach enough conversions in a set timeframe to have the right significance. We learned along the way that we then just don't test it at all.