Pausing a single variation in the middle of experiment (should I?)

[#1]

Hello,

We are currently running an experiment that has three variations. One of the variations ended up with a very low conversion rate and eventually deemed a "loser" by Optimizely. Being that we still want to receive conclusive results on the other versions, we want to keep the test going but we also don't want to continue to send traffic to the losing variation.

Would it be a bad practice to pause the individual variation and keep the rest of the experiment running? Or should we discontinue the experiment and start a new one without the loser? I don't want to throw away this experiment that has been running for about a week when we don't have conclusive results for the other versions. However, I am hesitant to send traffic to a page that is clearly a bad converter.

Any advice would be appreciated.

Thanks!
Noah

[#2]

Hi Noah,

Great question and thanks for posting in Optiverse!

I have found myself in this situation several times. Here is the best practice:

Unless you are certain that something is wrong, try not to make decisions about pausing variations until after your experiment has reached the sample size that you calculated before starting the experiment. Once you've reached the sample size you can pause the loser and decide to recalculate the sample size for the remaining two variations. Extending the sample size of the experiment may help you bring the statistical tie into significance.

Good luck!

[#3]

Adding to Kyle's response:

If you *do* decide to pause a variation, keep in mind that its conversion rates will still be updated if the test is running. At that point the variation's results should be ignored because they are not accurate.

Why? Because you already have a bunch of visitors bucketed into that variation, and if they convert at a later time (while the test is still running) then Optimizely will still count that conversion. The sample size, however, will remain constant because the variation is paused and no new visitors are being bucketed into it. As a result of this, the conversion rate may seem to go way up, and you may start thinking "wow, this variation wasn't as bad as I thought!"

In summary, if you do end up pausing a variation, take note of the results at that time and ignore subsequent results for that variation.

[#4]

Great addition, Greg. That can and will happen.

[#5]

Thanks Greg - Quick followup:

If i paused a variation in the middle of an experiment yesterday, then today I filter my results to show only today....why does the paused variation show any results at all...it should not have been running right?

[#6]

Once a visitor is bucketed into a variation, they'll continue to see that variation for as long as the experiment is running... Even if you stop allocating traffic to that particular variation (ie, if you "pause" the variation). If that visitor completes any goal, that goal will be credited to the variation they're in, even if it's not receiving new traffic.

In other words, "pausing" a single variation does not stop it from running, it only stops it from receiving new traffic.

Imagine a visitor, John, visits your site on Day 1 and is bucketed into variation B. At the end of Day 1 you stop allocating traffic to variation B. Then John comes back on Day 2. Assuming he did not delete his cookies, he will still see variation B. John might also complete a goal on Day 2, while seeing variation B.

Now you're looking at your results for Day 2. Even though variation B is not receiving *new* traffic, it will still show there was a conversion on Day 2, because of John.

Hope that helps!

[#7]

got it! thanks a bunch!

ryan

[#8]

Thank you all for the replies! This is very useful.

Have a good one!
Noah