Rich-text Reply

Measuring Return Visits in Optimizely

cubelodyte 01-07-15
Accepted Solution

Measuring Return Visits in Optimizely

One of the goals for an upcoming experiment is to increase return visits. 

We know that Google Analytics can examine this to some degree, but their sample rate is fairly sparce.

 

Does anyone know if there is a direct way in Optimizely results to measure increase or decrease in return visits due to an experiment variation?

 

I've searched the KB and the only references I can find for "Return Visit" talk about increase or decrease, but I'm not finding any mention of how it would be measured.

 

I suppose I could create a custom event JS that fires based on evaluation of the Optimizely object.

If anyone has already done this, I'd love to see the code as save some resource time.

 

My preference would be a built-in feature of the product.

Scott Ehly
Manager of Site Optimization
sehly@rentpath.com

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw
greg 01-07-15
 

Re: Measuring Return Visits in Optimizely

I'm not aware of any built-in feature that would do this. I must ask, though... Why not do this with Google Analytics? It would be really trivial, using the Frequency report. By saying the "sample rate is sparse," do you mean there are not many visitors? If so, wouldn't it still be true for Optimizely?

Level 2
Brian_Abad 01-07-15
 

Re: Measuring Return Visits in Optimizely

Hey @cubelodyte ,

 

If you'd like to track returning visitors we can utilize the Global JavaScript and Optimizely custom event goals to accomplish this.

 

Within the Global JavaScript, we will look for a cookie we drop in the user's browser once they have been bucketed into the Optimizely experiment. If this cookie is present, upon the visitors return we fire off a custom event goal that denotes they are a returning visitor. We will also drop this cookie that we're looking for within the Global JavaScript of the experiment. 

 

Here's the complete run down of what you'll want to do:

 

  1. WIthin the Global JavaScript of your experiment enter the following code: 
    window.optimizely = window.optimizely || [];
    if(document.cookie.indexOf('returning_visitor') > -1) {
    	window.optimizely.push(['trackEvent', 'returning_visitor']);
    }
    
    function setCookie(c_name,value,exdays,c_domain) {
        c_domain = (typeof c_domain === "undefined") ? "" : "domain=" + c_domain + ";";
        var exdate=new Date();
        exdate.setDate(exdate.getDate() + exdays);
        var c_value=escape(value) + ((exdays==null) ? "" : "; expires="+exdate.toUTCString());
        document.cookie=c_name + "=" + c_value + ";" + c_domain + "path=/";
    }
    
    setCookie('returning_visitor', 'true', 3650);

     

  2. Then, set-up a custom event goal with the Custom Event to Track as returning visitor. More on custom event goal set-up can be found here: https://help.optimizely.com/hc/en-us/articles/200039925-Custom-event-goals#add

More on Global JavaScript can be found here: https://help.optimizely.com/hc/en-us/articles/200039855-Global-JavaScript

---
Brian Abad
Manager, Technical Support
Customer Success
Optimizely
cubelodyte 01-07-15
 

Re: Measuring Return Visits in Optimizely

We are using Google Analytics currently. However, when I say the sample rate is sparse, it refers to the fact that GA puts a message at the top of the report:

"This report is based on 2.98% of sessions".
Scott Ehly
Manager of Site Optimization
sehly@rentpath.com

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw
cubelodyte 01-07-15
 

Re: Measuring Return Visits in Optimizely

[ Edited ]

Thanks, Brian.
That's exactly what I was looking for. You've saved me a lot of research.

Scott Ehly
Manager of Site Optimization
sehly@rentpath.com

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw
cubelodyte 01-12-15
 

Re: Measuring Return Visits in Optimizely

[ Edited ]

Well...
I may have overstated things a little.


It's not "exactly" what I was looking for, but it's on the right track.

 

One thing I'm trying to avoid is code in the Global JS that would be repeated for all experiments but require individual tweaking for each Experiment.

 

Brian,
The problem I see initially with your suggested code (with respect to my purposes) is that it will mark someone as a return visitor if they come in assigned to a new Experiment. My intention for the Goal is to measure whether or not a specific variation increases return visits.


Over time, if this code ends up in many or all new Experiments, we will likely begin seeing visitors return with the cookie set from a previous Experiment.

 

The immediate solution I see for this is to have the cookie contain the experiment ID and modify the code to insert the ID when writing and check for it during evaluation. I'm guessing that this can be done by looking at the optimizelyBuckets cookie. 

 

Example:

var optlyCookie = document.cookie.match("optimizelyBuckets=([^;]*)");

This is simple when it comes to writing the returning_visitor cookie, but l'm not sure how to determine which Experiment (if there are multiples) is the one from which the code is being executed without hardcoding the Experiment ID into the Global JS of each Experiment.

 

I was looking through the log today and saw 

Optimizely / Info / New vs returning: returning [time 20 +0]

 Does this mean there is something in the Optimizely object that determines New vs Returning?

 

If so, I'm guessing it's a generic check as well. In other words, it's probably just a matter of looking for an existing Optimizely cookie to see if the visitor has been there before without regard to the specific Experiment.

 

Additional Observation:

A second, and more important consideration is that our Experiments generally target multiple pages or even the entire site. I believe the code you presented will set off the event on the second page visit. I ultimately need something that can determine that a visitor has come back to the site in a new session.

 

Scott Ehly
Manager of Site Optimization
sehly@rentpath.com

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw
cubelodyte 01-12-15
 

Re: Measuring Return Visits in Optimizely

[ Edited ]

Ok, so here's what I finally did:

 

We already apply a specific custom code audience to each Experiment that ensures the Experiment is running exclusively. Because we run most Experiments on multiple pages or even the whole site, we need to keep visitors from being assigned to more than one Experiment at a time. 

 

So, since this Audience code already contains a specific reference to the Experiment ID and then compares that ID to the Experiment IDs in the bucket, I've added a check for a session cookie.

 

If the Experiment passes the exclusion check and there's not a session cookie already present, I fire the return_visit event and create the session cookie.

 

Example (new code in GREEN):

var currExpArray=[]; //Array of Experiment IDs for CURRENTLY ACTIVE Experiments
var optyExpArray = Object.keys(optimizely.allExperiments); //array of all experiments in project
for (var i = 0; i < optyExpArray.length; i ++) {
if (typeof(optimizely.allExperiments[optyExpArray[i]].enabled) !== 'undefined') //check for enabled property
currExpArray.push((optyExpArray)[i]);
}

var curExperiment = "0123456789"; //Get from URL in editor
var optlyCookie = document.cookie.match("optimizelyBuckets=([^;]*)"); //Get currently assigned experiments

var groupName = "__groupA";

if (!optimizely[groupName]){
var regexMatch = currExpArray.join("|");
optimizely[groupName] = (optlyCookie && optlyCookie[1].match(regexMatch)) ? optlyCookie[1].match(regexMatch)[0] : null;
}

//groupName now contains ID from expArray list, previous other experiment target code result or it's empty
//If empty, choose random from currExpArray

optimizely[groupName] = optimizely[groupName] || currExpArray[Math.floor(Math.random()*currExpArray.length)];

//Check for returning visitor
var sessionCookieName = 'opt' + curExperiment;
var optSessCookie = document.cookie.match(sessionCookieName);

//Check if new session and current Experiment matches random exclusion

if (!optSessCookie && optimizely[groupName]==curExperiment) {
//Check for curExperiment in Optimizely Buckets cookie
if (optlyCookie && optlyCookie[1].match(curExperiment)){
window.optimizely.push(['trackEvent', 'return_visit']);
}
//Create session cookie
var cText = sessionCookieName + '=true';
cText = cText + ';path=/';
document.cookie = cText;
}
//Boolean for Audience
(optimizely[groupName] == curExperiment);

 We're already in the practice of updating the Experiment ID in this code for each new Experiment. Aside from that one customization to update the ID, we still have a single code block that can be duplicated in any new Experiment. 

 

...And, as a bonus...it can be done without altering our standard Global JS.

Scott Ehly
Manager of Site Optimization
sehly@rentpath.com

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw