Rich-text Reply

EXCLUSIVE PROJECT CODE: VARIATIONS ON A THEME

cubelodyte 04-11-16

EXCLUSIVE PROJECT CODE: VARIATIONS ON A THEME

[ Edited ]

 

Different Business Model

First off, we would like to thank the Optimizely technical staff for their initial pioneering work on mutually exclusive project code. This innovation has allowed us to move away from Audience level codingThis audience code requires a great deal more maintenence (including experiment IDs and changing the arrays when new experiments are added). 

 

Our sites tend to differ in nature from many other sites with respect to both visitor funnel and KPI. One of the most significant characteristic differences is the fact that behavior on one page can have a profound effect on the behavior of another. Because of this, our experiments tend to run across the entire site rather than on any one specific page; this is true even if the variations themselves only affect a single page. 

 

As a consequence, we have found that running our experiments as mutually exclusive has become the norm rather than the exception. In our testing model, we only want an experiment to interact with another if that other experiment is being used as a Hotfix or site-wide change that is desired outside of our standard release cycle.

 

Building on the Exclusive Project code provided by Optimizely [here], we've made a few revisions:

  1. An experiment is considered exclusive by default and only allowed to run simultaneously with experiments that are marked as inclusive in the name (the opposite of how the provided code evaluates the experiment names).
  2. During the QA phase of our testing, it is necessary to force a variation and we need this forced assignment to remain in place when navigating away from the initial target page.

Exclusive by Default

The first modification we made is to reversed the trigger for identifying an experiment as exclusive or non-exclusive (henceforth to be labled here as "inclusive"). This allows us to keep our experiment names cleaner and more manageable be eliminating the need to add a prefix to the name of every non-exceptional experiment. 

 

In the following code example, we also simplified the selection process by removing the [group] distinction; a construct that we have found unnecessary for our more limited testing needs. 

 

var noTagArray = ["^hotfix","^preview","^notag"];
var _regExMatch = noTagArray.join("|");

function bIsInclusive(expName){
    if (expName.match(_regExMatch)) return true;
    return false;
}
function getRunningExperiments_inclusive(bGetInclusive) {
    var running_experiments = [];
    var bInclusive = false;
    for (var exp_id in DATA.experiments) {
        var exp = DATA.experiments[exp_id];
        if (exp.enabled && DATA.experiments[exp_id].name !== undefined) {
            bInclusive = bIsInclusive(DATA.experiments[exp_id].name);
            if (bInclusive === bGetInclusive) {
                running_experiments.push(exp_id);
            }
        }
    }
    return running_experiments;
}

The original project code builds 2 arrays to house the exclusive and inclusive experiement IDs. In the code sample above, this is done by passing a boolean value into the function getRunningExperiments_inclusive

 

Note: the original function was based on 'exclusive' so the boolean values are reversed to go along with the name.

 

To create your own prefixes for identifying inclusive experiements, you can modify the array here:

 

var noTagArray = ["^hotfix","^preview","^notag"];

 

You can also change the RegEx or use indexOf if you choose to place the identifying string somewhere other than at the beginning of the experiment name. 

 

Forcing a Variation Across Pages

Another aspect of the original project code is that it intentionally eliminates forced experiment variations from it's list of running experiments:

 

function getRunningExclusiveExperiments(exclusive) {
    var running_experiments = [];
    for (var exp_id in DATA.experiments) {
        var exp = DATA.experiments[exp_id];
        if (exp.enabled && window.location.search.indexOf("x" + exp_id) === -1 && DATA.experiments[exp_id].name !== undefined) {
            var groupIndex = DATA.experiments[exp_id].name.toLowerCase().indexOf("[Group_".toLowerCase());
            var meIndex = DATA.experiments[exp_id].name.toLowerCase().indexOf("[ME]".toLowerCase());
            if ((groupIndex > -1 && exclusive) || (meIndex > -1 && exclusive)) {
                running_experiments.push(exp_id);
            } else if (groupIndex == -1 && meIndex == -1 && !exclusive) {
                running_experiments.push(exp_id);
            }
        }
    }
    if (exclusive) {
        window.running_exclusive = running_experiments;
    } else {
        window.running_non_exclusive = running_experiments;
    }
    return running_experiments;
}

The net result is that a QA preview will appear for any page load that includes the query string, but the assignment will not be bucketed for subsequent page views. In our modified example, the forced experiment variation will continue for the QA tester until cleared.

 

Note: Our standard practice for QA is to clear all cache and cookies before and after each preview. If you use variation forcing in consumer-facing experiments, you will want to modify accordingly.  

 

Complete Code Block

For context, our entire project code block for running experiments as exclusive or inclusive can be found below:

 

var noTagArray = ["^hotfix","^preview","^notag"];
var _regExMatch = noTagArray.join("|");

function bIsInclusive(expName){
    if (expName.match(_regExMatch)) return true;
    return false;
}

function getParameterByName(name) {
    name = name.replace(/[\[]/, "\\[").replace(/[\]]/, "\\]");
    var regex = new RegExp("[\\?&]" + name + "=([^&#]*)"),
        results = regex.exec(location.search);
    return results === null ? "" : decodeURIComponent(results[1].replace(/\+/g, " "));
}
var docCookies = {
    getItem: function(sKey) {
        return decodeURIComponent(document.cookie.replace(new RegExp("(?:(?:^|.*;)\\s*" + encodeURIComponent(sKey).replace(/[\-\.\+\*]/g, "\\$&") + "\\s*\\=\\s*([^;]*).*$)|^.*$"), "$1")) || null;
    },
    setItem: function(sKey, sValue, vEnd, sPath, sDomain, bSecure) {
        if (!sKey || /^(?:expires|max\-age|path|domain|secure)$/i.test(sKey)) {
            return false;
        }
        var sExpires = "";
        if (vEnd) {
            switch (vEnd.constructor) {
                case Number:
                    sExpires = vEnd === Infinity ? "; expires=Fri, 31 Dec 9999 23:59:59 GMT" : "; max-age=" + vEnd;
                    break;
                case String:
                    sExpires = "; expires=" + vEnd;
                    break;
                case Date:
                    sExpires = "; expires=" + vEnd.toUTCString();
                    break;
            }
        }
        document.cookie = encodeURIComponent(sKey) + "=" + encodeURIComponent(sValue) + sExpires + (sDomain ? "; domain=" + sDomain : "") + (sPath ? "; path=" + sPath : "") + (bSecure ? "; secure" : "");
        return true;
    },
    removeItem: function(sKey, sPath, sDomain) {
        if (!sKey || !this.hasItem(sKey)) {
            return false;
        }
        document.cookie = encodeURIComponent(sKey) + "=; expires=Thu, 01 Jan 1970 00:00:00 GMT" + (sDomain ? "; domain=" + sDomain : "") + (sPath ? "; path=" + sPath : "");
        return true;
    },
    hasItem: function(sKey) {
        return (new RegExp("(?:^|;\\s*)" + encodeURIComponent(sKey).replace(/[\-\.\+\*]/g, "\\$&") + "\\s*\\=")).test(document.cookie);
    },
    keys: /* optional method: you can safely remove it! */
        function() {
        var aKeys = document.cookie.replace(/((?:^|\s*;)[^\=]+)(?=;|$)|^\s*|\s*(?:\=[^;]*)?(?:\1|$)/g, "").split(/\s*(?:\=[^;]*)?;\s*/);
        for (var nIdx = 0; nIdx < aKeys.length; nIdx++) {
            aKeys[nIdx] = decodeURIComponent(aKeys[nIdx]);
        }
        return aKeys;
    }
};

function getRunningExperiments_inclusive(bGetInclusive) {
    var running_experiments = [];
    var bInclusive = false;
    for (var exp_id in DATA.experiments) {
        var exp = DATA.experiments[exp_id];
        if (exp.enabled && DATA.experiments[exp_id].name !== undefined) {
            bInclusive = bIsInclusive(DATA.experiments[exp_id].name);
            if (bInclusive === bGetInclusive) {
                running_experiments.push(exp_id);
            }
        }
    }
    return running_experiments;
}
function expArrayToJSON(arr, item, exclusive_only) {
    var result = {};
    if (!exclusive_only) {
        for (var exp_id in arr) {
            if (exp_id != item) {
                result[exp_id] = "0";
            }
        }
    } else {
        for (var exp_id in arr) {
            if (exp_id != item && arr[exp_id].name.toLowerCase().indexOf("[EXCLUSIVE]".toLowerCase()) > -1) {
                result[exp_id] = "0";
            }
        }
    }
    return result;
}
function log(msg) {
    window['log_arr'] = window['log_arr'] || [];
    window['log_arr'].push(msg);
}
window.print_log = function() {
    window['log_arr'] = window['log_arr'] || [];
    console.log(window['log_arr'].join('\n '));
};
function groupFlow(items, exclusive_items) {
    for (var key in exclusive_items.groups) {
        var groupKey = exclusive_items.groups[key];
        //check for group holdout
        if(groupKey.holdout !== false && (Math.floor(Math.random() * 100) > 95)) {
            items.push(groupKey.holdout);
        //else push group experiments
        } else {
            var randomNum = Math.floor(Math.random() * (groupKey.exps.length + groupKey.me.length));
            if(randomNum < groupKey.exps.length) {
                //push all group experiments if non 'me' is randomly selected
                items = items.concat(groupKey.exps);
            } else {
                //push one 'me' experiment if 'me' is randomly selected
                items.push(groupKey.me[(randomNum-groupKey.exps.length)]);
            }
        }
    }
    return items;
}
function pickExperiment(pick_exp) {
    var topdomain = location.hostname.split(".");
    topdomain = "." + topdomain[topdomain.length - 2] + "." + topdomain[topdomain.length - 1];
    //get lists of inclusive and exclusive experiments
    var inclusive_items = {
        allExp: getRunningExperiments_inclusive(true)
    };
    window.running_inclusive = inclusive_items;

    var exclusive_items = {
        allExp: getRunningExperiments_inclusive(false)
    };
    window.running_exclusive = exclusive_items;

    //check if experiment exists
    var items = [];
    exclusive_items = sortGroups(exclusive_items);
    if ((exclusive_items.allExp.indexOf(pick_exp) > -1) || (inclusive_items.allExp.indexOf(pick_exp) > -1)) {
        items.push(pick_exp);
    //check 'ME' holdout chance
    } else if (exclusive_items['me'] !== undefined && exclusive_items['me']['holdout'] !== undefined && Math.floor(Math.random() * 100) > 95) {
        items.push(exclusive_items['me']['holdout']);
    //check if groups exist, if not choose me
    } else if(exclusive_items['groups'] === undefined && exclusive_items['me'] !== undefined) {
        items.push(exclusive_items['me']['exps'][Math.floor(Math.random() * exclusive_items['me']['exps'].length)]);
    //check if me exist, if not go through group flow
    } else if(exclusive_items['me'] === undefined && exclusive_items['groups'] !== undefined) {
        //do group flow
        items = groupFlow(items, exclusive_items);
    } else if(exclusive_items['groups'] !== undefined && exclusive_items['me'] !== undefined) {
        var groupExps = exclusive_items.allExp.length - exclusive_items.me.exps.length;
        if(Math.floor(Math.random() * exclusive_items.allExp.length) < groupExps) {
            //do group flow
            items = groupFlow(items, exclusive_items);
        //push an me experiment
        } else {
            items.push(exclusive_items['me']['exps'][Math.floor(Math.random() * exclusive_items['me']['exps'].length)]);
        }
    }
    items = items.concat(inclusive_items.allExp);
    docCookies.setItem('optimizelyExp', items, 2000 * 86400000, "/", topdomain);
    return items;
}
function sortGroups(items) {
    for (var i = 0; i < items.allExp.length; i++) {
        var exp = DATA.experiments[items.allExp[i]];
        var expName = exp.name.toLowerCase();
        //experiment is in a group
        if(expName.indexOf("group_") > -1) {
            //set group object if it doesn't exist
            if(items['groups'] === undefined) { 
                items['groups'] = {};
            }
            //set specific group if it doesn't exist
            if(items['groups'][expName.charAt(7)] === undefined) {
                items['groups'][expName.charAt(7)] = {'holdout': false, 'me': [], 'exps': [] };
            }
            //check if holdout first
            if(expName.indexOf('holdout') > -1) {
                items['groups'][expName.charAt(7)]['holdout'] = items.allExp[i];
            //check if mutually exclusive within groups
            } else if(expName.indexOf('me') > -1) {
                items['groups'][expName.charAt(7)]['me'].push(items.allExp[i]);
            //push to all group exps otherwise
            } else {
                items['groups'][expName.charAt(7)]['exps'].push(items.allExp[i]);
            }
        //experiment is an me experiment
        } else {
            //if undefined, create object
            if(items['me'] === undefined) { 
                items['me'] = { 'holdout': false, 'exps': [] };
            }
            if(expName.indexOf('holdout') > -1) {
                items['me']['holdout'] = items.allExp[i];
            } else {
                items['me']['exps'].push(items.allExp[i]);
            }
        }
    }
    return items;
}
function updateBuckets(items) {
    window['optimizely'] = window['optimizely'] || [];
    log("Mutual exclusion chose experiment " + items);
    for (var exp_id in DATA.experiments) {
        if (items.indexOf(exp_id) === -1 && window.location.search.indexOf("x" + exp_id) === -1) {
            DATA.experiments[exp_id].enabled = false;
            log("Disable experiment " + exp_id);
 //       } else {
 //           console.log('Active - ' + exp_id + ': ' + DATA.experiments[exp_id].name);
        }
    }
}
if (typeof DATA != 'undefined') // DATA object isn't defined in the Optimizely editor but is when the code runs on the page
{
    var force_exp = getParameterByName("optimizely_exclusive_force");
    var exp = docCookies.getItem('optimizelyExp');
    if(exp !== null && exp.indexOf(',') > -1 && force_exp == "") {
        exp = exp.split(',');
        for(var i = 0; i < exp.length; i++) {
            if(!(DATA.experiments.hasOwnProperty(exp[i]) && DATA.experiments[exp[i]].enabled)) {
                var exp = pickExperiment(force_exp);
            }
        }
    } else {
        if ((!exp || !(DATA.experiments.hasOwnProperty(exp) && DATA.experiments[exp].enabled)) || force_exp != "") {
            var exp = pickExperiment(force_exp);
        }
    }
    window.opt_expid = exp;
    updateBuckets(exp);
}

 

Scott Ehly
Manager of Site Optimization
sehly@rentpath.com

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw

Re: EXCLUSIVE PROJECT CODE: VARIATIONS ON A THEME

Hi Scott,

Thanks so much for posting this here! We try to provide some custom code solutions that are applicable for as many use cases as possible. It's great to see that you and others like you take those code samples and tweak them for your own specific use cases.

Many kudos!

Best wishes,
Nils