Rich-text Reply

Changes in Production Code Affecting Existing Experiments

cubelodyte 06-26-14

Changes in Production Code Affecting Existing Experiments

Occasionally, the regular release cycle for a site introduces code changes that can break existing Optimizely experiments. There are several pain points associated with adjusting an experiment to accommodate these changes in the site. The issue revolves primarily around the experience of currently assigned visitors.


Pausing the Experiment

At present, our chosen process for dealing with site releases that break variations is to:


  1. Pause the experiment just prior to release
  2. Release the new site code
  3. Modify broken variations
  4. Start experiment again


With the new Project setting “Exclude draft and paused experiments”, we have effectively created 2 scenarios through which this solution can run. Each has its own drawback.


  • If the project is not set to “Exclude draft and paused experiments”, then new visitors will not be assigned to a variation, but existing visitors will.
  • If this setting is checked, then existing visitors will have their experience change on them (possibly in the middle of a site visit).


The second option listed is clearly preferable, given the fact that the variation is temporarily broken. However, either condition leads to a disrupted experience and “noise” in the results.


As a rule, we tend to throw out the first and last day of any experiment results to reduce the effect of participants for whom the experience has changed mid-session. Introducing these conditions into the middle of an experiment creates noise in the results that can’t be filtered out so easily.


Does anyone have a strong, effective plan for handling situations in which newly released production code breaks existing experiments?


Scott Ehly
Manager of Site Optimization

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw
Josh 06-27-14

Re: Changes in Production Code Affecting Existing Experiments

Hey Cubelodyte,

Ideally code changes shouldn't break your Optimizely experiments and you should QA with Optimizely in mind.

Do you have a dev environment that you code in prior to release?

If so I would suggest that you:
1. Create a separate Optimizely Project for your dev environment and install that code snippet
2. Duplicate all of your experiments from the production environment into the dev environment.
3. Teach your QA team how to force variations in Optimizely
Level 2