Rich-text Reply
Highlighted

Do you run perpetual experiments you leave experiments?

vanessa_krumb 02-12-16

Do you run perpetual experiments you leave experiments?

Hi Folks!

 

I've seen people run Experiments, then just leave the winner running, rather than actually change the website. In particular I've seen this where a dev team has a long backlog, but the CRO team wants the winning page to be implemented.

 

Do you ever do this? How common do you think this is?

---------------------------------- - - - - - ➳
Vanessa

Revere.ai Monitor your Optimizely experiments

Micael 02-15-16
 

Re: Do you run perpetual experiments you leave experiments?

Hi Vanessa!

 

As much as possible we, in our organization, prefer to have all winning experiments implemented in the site as soon as it goes green as we run dozens of experiments at a time. These unarchived experiments contibute to the overall weight of the Optimizely script and in the long run negatively affect loading time for our users but we still, in some cases, leave some of them running because of long backlog issues with the dev team.

 

We have a pretty huge testing limit plan with Optimizely so leaving winning experiments to run for an extended amount of time doesn't really cost us a thing but I can see how this can be a problem for other teams with a limited testing plan as running experiments contirbute to their monthly testing allocation.

 

Aside from leaving winning running experiments in production during times when the dev team's backlog is packed, we also use Optimizely to deploy and keep hot fixes. 

Mica Diaz de Rivera
Level 2

Re: Do you run perpetual experiments you leave experiments?

Thanks Micael! 

 

Add on question - do you define 'as soon as it goes green' as when you have a single winning variation for your main goal in the experiment, or when you have reached confidence level for all variations in the experiment, for your main goal?

---------------------------------- - - - - - ➳
Vanessa

Revere.ai Monitor your Optimizely experiments

Micael 02-16-16
 

Re: Do you run perpetual experiments you leave experiments?

[ Edited ]

Good question! It's usually the latter - when we have reached the confidence level for all variation in the experiment for our main goal.

 

For our organization, we set up a testing time frame (usually a week or two) for most of our tests and most of the time we either get no statistical gains on any of the variations or we reach a confidence level for all varation in the experiement for our main goal. If it's the former, we pivot. If it's the latter, we recommend that it be inserted in the product pipeline to be worked on by the dev team.

 

There are some few instances though that only one variation gets to go green and reach a level of confidence at the end of the time frame we alloted. If this happens, yes, we recommend the said variation and forgo with the other ones that have yet to reach significance.

Mica Diaz de Rivera
Level 2

Re: Do you run perpetual experiments you leave experiments?

[ Edited ]

I agree with Mica as well, but I will say that this is fairly common to a certain extent. We have a fairly long lead time on dev backlog, so I typically duplicate the experiment when a winner is found, pause the old, and launch the new one which has 100% of traffic going to the winning variation. I will let this run for the one to two weeks while the dev team implements the changes on production. As Mica mentioned the Optimizely script can start to create page load increases, so I make sure to archive as many past experiments as possible and stagger my experiment schedule as to not create too many instances where winners are running and several experiments simultaneously.

-----

Daniel


Digital Marketing Manager - mindbodygreen
Optiverse User Group Leader - NYC