Rich-text Reply

Supporting customers during a/b tests?

mvreyes 09-30-15

Supporting customers during a/b tests?

Hi all,


I have a question about when tests are running. Some of the tests that we're doing affects navigation and we're running into an issue of how to support customers who are seeing a different version. We'll soon be doing more personalization and testing so I'm wondering if any of you had to deal with this situation?



Level 1

JDahlinANF 09-30-15

Re: Supporting customers during a/b tests?

It starts with communication.  They have to know what questions to ask to even realize that an experiment is in play and know how to identify which variation of which experiment the user is seeing.


I like to provide users (including customer care reps) code that allows them to switch to specific versions of the experiment.  That way they can see what the customer sees (once they identify which variation of the experiment the customer is seeing).


For this reason, we left "optimizely force" enabled (e.g., adding *optimizely_xNNNNNNN=1" to the URL to switch to the B variation of experiment NNNNNNN).


Alternatively, you could provide some javascript code that updates the user's cookie to a specific variation.  Here is a template you could use - you would need to fill in the values for each experiment and variation and provide documentation to the teams on how/when to use it, etc.

var myVariation = "B";           //Choose A or B here
var experimentId = experiment_ID_here ;  //EXPERIMENT NAME HERE 
var variationA = variation_ID_here ;    //A_VARIATION NAME
var variationB = variation_ID_here ;    //B_VARIATION NAME

switch(myVariation) {
	case "A":
		window['optimizely'].push(["bucketVisitor", experimentId, variationA ]);
	case "B":
		window['optimizely'].push(["bucketVisitor", experimentId, variationB ]);


Pauline 10-01-15

Re: Supporting customers during a/b tests?

We've run big tests in the past where the experience / booking flow might be impacted. 

Here are a few tips:


- Do user research before and run labs with your new design ( or in person labs with customers interviews). If you design a better experience, customers will like it better.

- Talk to customer service ahead of time so they can train customer care agents to answer any questions from customers

- Target your experiement only for new users first and expand it to repeat customers only once you have encouraging preliminary results.

- Start with a small traffic allocation (if you usually do 50/50 maybe start with 95/5 and slowly ramp up)

- When you look at results look at impact for new vs repeat users


Product Manager
Level 2