- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

# Uplift numbers changed after pausing one variation

**Shlomo**08-05-15

# Uplift numbers changed after pausing one variation

[ Edited ]- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report

We launched a test with 2 variations.

Initial results after 3 days showed variation 1 had 50% uplift and 77% signifigance.

I paused variation 2 as it was -2% .

After that the variation 1 has tanked drastically from 50% to only 9% increase.

Why did this happen?

Is there a corellation betwen pausing the second variation and the decrease?

Any ideas?

**MartijnSch**08-06-15

## Re: Uplift numbers changed after pausing one variation

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report

## Re: Uplift numbers changed after pausing one variation

[ Edited ]- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report

Hi!

Posted the same question a while back and @brianlang provided a perfect response.

Have a look at the discussion here: https://community.optimizely.com/t5/Strategy-Culture/Multiple-goals-and-variations-impact-on-false-d...

Hope this helps.

Best,

Marie

## Re: Uplift numbers changed after pausing one variation

[ Edited ]- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report

You probably haven't run the test long enough to be perfectly honest! Sometimes I find when pausing variations it can be better to start a totally new test, as all returning users will be continuing to see Variation 2 - potentially leading to confusing (at least at first glance) data.

Technical Product Manager - Conversion @ Secret Escapes

## Re: Uplift numbers changed after pausing one variation

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report

Volatility early on is normal and should be expected. I wrote a few responses in the following threads that could explain what you are observing - read through and let me know if you have any questions!

Volatile winner results in daily data

How would i report on these inconclusive results?

twitter.com/cometrico

## Re: Uplift numbers changed after pausing one variation

[ Edited ]- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report

Without knowing all the details, I agree with MARTIJNSCH and continually see it in our own testing that as your sample size increases, the volatility also tends to even out as a winner emerges with your set significance level. This is especially true with revenue based tests, where one variation could receive a larger order, early on. Over time, as your N increases, the true winner should emerge.