Subsequent Tests don't have similar conversion rates, why?
Hi, I need a little help explaing why conversion rates are different between tests.
We've run one successful test on our homepage with the winning variation converting around 2%. We then updated the homepage to include those winning changes. Afterwards, we lauched the next test, where the control is the winning variation from the previous test. However, the control is converting around .9% instead of 2% like before. Why is that? Thanks!
I would check the traffic sources first as that normally is where you would find the answer to this sort of problem. You could then look at seasonal traffic trends, are we at a lower performing period of a month now, compared to a high performing part of the month when you did your previous test?
Whilst other commenters have offered useful suggestions, I'd also like to add that it's possible that the test was called before reaching statistical significance.
I'd like to direct you to this sample size calculator, which helps decide how many visitors should have been included in each branch (variation) of your test before achieving a conclusive results.
To break down the terms:
Baseline conversion rate is your conversion rate for the original page
Minimum detectable effect is the % uplift you're seeing on the variation.
The calculator should also be set to "relative".
Once you've input the values, you'll be given the number of visitors requires for each variation before the results can be considered statistically significant. If this test had not reached the recommended number of visitors by the time you launched the winner, this may be the reason behind the lower-than-expected conversion rate.
If you've any follow-up questions, just let me know.
Your conversion rate was 2%. Good but, what about the error interval? if it was (2±1)% then you shouldn't be surprised at all. The C.R. itself doesn't make much sense if you don't keep an eye on its "error bars".
Other answers also explain what you are experiencing. It's just statistics.
Project Manager at 21DIAMONDS GmbH