Rich-text Reply

Uplift numbers changed after pausing one variation

Shlomo 08-05-15

Uplift numbers changed after pausing one variation

[ Edited ]

We launched a test with 2 variations.

Initial results after 3 days showed variation 1 had 50% uplift and 77% signifigance.

I paused variation 2 as it was -2% .

After that the variation 1 has tanked drastically from 50% to only  9% increase.

 

Why did this happen?

Is there a corellation betwen pausing the second variation and the decrease?

 

Any ideas?

Level 2

MartijnSch 08-06-15
 

Re: Uplift numbers changed after pausing one variation

I won't see there is a correlation, usually after 3 days and that also shows in the significance level of 77% (which is very low!, we usually run tests on a +95% significance) it's not by far clear yet what variant could win the test. If you then pause one of the variants the other two will just get more traffic and faster reach significance. In my experience what then usually happens is that the conversion rate rate also drops as you start sending more traffic to it.
marie 08-06-15
 

Re: Uplift numbers changed after pausing one variation

[ Edited ]

Hi!

Posted the same question a while back and @brianlang provided a perfect response.
Have a look at the discussion here: https://community.optimizely.com/t5/Strategy-Culture/Multiple-goals-and-variations-impact-on-false-d...

Hope this helps.

Best,

Marie

Level 2
td_evans 08-06-15
 

Re: Uplift numbers changed after pausing one variation

[ Edited ]

You probably haven't run the test long enough to be perfectly honest! Sometimes I find when pausing variations it can be better to start a totally new test, as all returning users will be continuing to see Variation 2 - potentially leading to confusing (at least at first glance) data.

Thomas Evans
Technical Product Manager - Conversion @ Secret Escapes

Level 2
brianlang 08-07-15
 

Re: Uplift numbers changed after pausing one variation

Volatility early on is normal and should be expected. I wrote a few responses in the following threads that could explain what you are observing - read through and let me know if you have any questions!

 

Volatile winner results in daily data

How would i report on these inconclusive results?

Brian Lang
twitter.com/cometrico
eOps 08-20-15
 

Re: Uplift numbers changed after pausing one variation

[ Edited ]

Without knowing all the details, I agree with MARTIJNSCH and continually see it in our own testing that as your sample size increases, the volatility also tends to even out as a winner emerges with your set significance level. This is especially true with revenue based tests, where one variation could receive a larger order, early on. Over time, as your N increases, the true winner should emerge.

Level 2