Rich-text Reply

Is it possible to only show one variant to a user when running multiple A/B-tests?

engelhardt 12-02-14

Is it possible to only show one variant to a user when running multiple A/B-tests?

Our conversion-process (as probably many others) consists of multiple pages. We run multiple A/B-tests at a time, but only one test per page.

 

Lets say we have one test for each page in our conversion process.

What i want to do is to exclude one user which ran into the variant of page 1 from every other test on the following pages, since this would corrupt the results.

Or in general: I want exclude one user from the variants of all following A/B-Tests after he ran into the variant of one test.

 

Is this possible? And if yes, does it work properly? how can i do this?

 

Thanks
Daniel

JDahlinANF 12-02-14
 

Re: Is it possible to only show one variant to a user when running multiple A/B-tests?

[ Edited ]

The easiest way to handle this, IMO, is to set up one experiment where each variation only targets one page of the funnel.

 

For example, if you have 3 steps in your checkout and want to run an A/B on each page:

 

Variation A targets Shipping Info

Variation B targets Payment Info

Variation C targets Order Review

 

To achieve this you would set URL targeting to include all 3 URLs and include code in the varitions that limit which URL the code affects.

e.g.

if (window.location.pathname.indexOf('shipping') > 0) {
  //Variation A's changes go here
}

 

This easily ensures that users are only in one version of the overall experiment. 

Unfortunately, this approach means that you would not be able to use mere participation in the experiment for your funnel metrics (Users in the B and C variations are in the experiment even though they did not actually reach their page of the experiment).

 

A slightly more complicated way to go would be to set up the overall experiment as 3 separate experiments (let's call them "shipping", "payment', and 'review") and use conditional javascript in each experiment that excludes the other two experiments' "B" variations (using the IDs of the variations from the diagnostic editor).  Like this:

 

First - set up the experiments:

Experiment Shipping with variations A and B

Experiment Payment with variations A and B

Experiment Review with variations A and B

 

Second - use the diagnostics report to find the ID numbers of each variation:

Experiment Shipping with variations A (111111) and B (222222)

Experiment Payment with variations A (333333) and B (444444)

Experiment Review with variations A (555555) and B (666666)

 

Third - add conditional javascript to each experiment that excludes the "B" versions of the other experiments:

For Experiment Shipping, it could look like this (the function at the top checks to see which variations a user is in, the second line from the bottom is where you place the IDs you want to check against).  In this case we want to exclude users who are in either variation, so where the function returns a "true", we want the audience targeting to return "false" (to exclude them):

function checkVariation(variantID) {
	var OptSegment = (function(){
		var name = "optimizelyBuckets";
		var ID = variantID;
		var cookie = document.cookie.match(name+"=([^;]*)") || "";
		if (cookie != null) {
			var decodedCookie = decodeURIComponent(cookie[0]);
			var value = decodedCookie.match('"([^,]*)":"'+ID+'"');
			return value;
		} else {
			return null;
		}
	})();
	var optlyVal = false;
	if (OptSegment != null) {
		optlyVal = true;
	}
	return optlyVal;
}


(function(){
  return !checkVariation('444444') || !checkVariation('666666');
})();

The tricky part here will be getting enough sample size for your 3rd experiment...

Using a 50-50 split on each experiment would mean that only 25% of the users who hit Payment would be placed in the Payment-B varitation and only 12.5% of users who reach the review page would be in the Review-B variation.

 

nolanmargo 12-02-14
 

Re: Is it possible to only show one variant to a user when running multiple A/B-tests?

[ Edited ]

Before heading down the elegant route suggested by nap0leon, I'd suggest familiarizing yourself with this awesome article from Optimizely, which discusses the Simultaneous Testing conundrum, especially as it relates to introducing "noise" into an experiment.  

https://help.optimizely.com/hc/en-us/articles/200064329-Simultaneous-Testing-Running-two-different-t...

engelhardt 12-04-14
 

Re: Is it possible to only show one variant to a user when running multiple A/B-tests?

Thanks a lot for this detailed answer! Will give it a try