Rich-text Reply

How to change % variant allocations between different environments

isaactyler 06-12-18

How to change % variant allocations between different environments

Is there a way to use different variant % allocations between environments?


My team currently has been creating 2 copies of each experiment for each environment (one for "test" and one for "production"). Recently, my team stumbled across an article that encourages creating a single experiment and having that apply to different environments ("test" and "production") as it's more manageable. The only problem is that once doing so, changing the variants for the experiment would change it for all the environments running that experiment. 


It would be great to be able to test and allocate 100% of a traffic for a test environment without having it affect the production environment.


Is there a way to accomplish doing so? Or are there other best practices regarding how to handle testing specific variants in one environment and not affect the rest?

JasonDahlin 06-13-18

Re: How to change % variant allocations between different environments

[ Edited ]

I built our program using 2 versions of each experiment. One for test and one for production.
In our experience you can never have only one version of an experiment...
- QA cannot (easily) run automation on an inconsistent experience
- Not all tests can be active in all local/DEV/QA/UAT environments and sometimes need to be active but 0% so that only the DEV assigned to the task has access
- What if you need to test minor tweaks to the experiment? Do you really want to test in production?

So, we always start with a "QA-Experiment name here" and use Audiences to control which environments it runs in. Depending on what is being tested and the environment, this experiment may be 100% control and the DEV folks opt into the B group using the force parameters (optimizely_x=variation id here).

Once the experiment has passed QA and received Business Owner signoff, I duplicate the experiment, change the name to the production name (the name we want tracked in analytics), assign the tracking variable (integrations), and set the audience to "production".

With this setup, we can modify the test experiment as much as we need to. Whether it is to tweak it or to move it out of a lower level environment but keep it available in UAT, etc.

--Jason Dahlin
Analytics and Testing Guru Smiley Happy

Experimentation Hero