Uplift numbers changed after pausing one variation
We launched a test with 2 variations.
Initial results after 3 days showed variation 1 had 50% uplift and 77% signifigance.
I paused variation 2 as it was -2% .
After that the variation 1 has tanked drastically from 50% to only 9% increase.
Why did this happen?
Is there a corellation betwen pausing the second variation and the decrease?
Posted the same question a while back and @brianlang provided a perfect response.
Have a look at the discussion here: https://community.optimizely.com/t5/Strategy-Cultu
Hope this helps.
You probably haven't run the test long enough to be perfectly honest! Sometimes I find when pausing variations it can be better to start a totally new test, as all returning users will be continuing to see Variation 2 - potentially leading to confusing (at least at first glance) data.
Technical Product Manager - Conversion @ Secret Escapes
Volatility early on is normal and should be expected. I wrote a few responses in the following threads that could explain what you are observing - read through and let me know if you have any questions!
Without knowing all the details, I agree with MARTIJNSCH and continually see it in our own testing that as your sample size increases, the volatility also tends to even out as a winner emerges with your set significance level. This is especially true with revenue based tests, where one variation could receive a larger order, early on. Over time, as your N increases, the true winner should emerge.