Rich-text Reply

What is the most sophisticated integration done with the Optimizely API?

stefank 11-25-14

What is the most sophisticated integration done with the Optimizely API?

Hi,

 

I'm looking for inspiration as to what can be achieved when you tightly integrate with the Optimizely API. Are there any good examples/case studies?

 

In particular I'm looking for examples where people have integrated optimizely with their own in house experimentation, or reporting/analytics collection frameworks. It would be nice to hear what the pitfalls are, and what the strengths are.


Pointers, suggestions or whitepapers appreciated.

 

Thanks in advance.


Cheers,


Stefan

Level 2

Joel_Balmer 11-25-14
 

Re: What is the most sophisticated integration done with the Optimizely API?

Hi Stefan,

What sort of things are you interested in?

There are cool things that could be created with the results API

http://developers.optimizely.com/rest/#get-experiment-results-18

For instance, you could create a dashboard that shows a neat overview, or more insights that you would like to see, for all experiments at once.

I'm also interested to know what ideas people have / what people have come up with!
cubelodyte 11-25-14
 

Re: What is the most sophisticated integration done with the Optimizely API?

[ Edited ]

Stephan, 

We've been integrating the Optimizely solution with our own data collection from the start (a little over 2 years ago). To be honest, I'd love to hear some better ideas for in-house reporting, but I can relate some things we've done on the Optimizely side.

 

In House Data

To begin with, we track nearly every user event from page request to exit and all the clicks in between. Everything the user does is sent back to an SQL data warehouse where we can analyze to our hearts content.

[This is where I'd love to hear some better reporting ideas*]

 

Tagging

The first step to integrating our Optimizely experiments with this data is to "tag" the events with experiment name and variation name. This allows us to parse out visits within the experiment from those outside the experiment. Our tagging routines build all of the site/page specific identifiers very early in the document load.

 

One of the hooks into our code for building the identifiers is the use of meta tags. This is our way in. I've placed into the Global js of every experiment, a generic code block that will grab the names of the Experiment and Variation from the Optimizely data object. The script will build a string based on the names associated with activeExperiments, and insert them as meta tags at the top of the HTML.

 

The code below is an example:

 

var oData = window.optimizely.data;
var activeExperiments = oData.state.activeExperiments;

if (oData.state.activeExperiments && oData.state.activeExperiments.length) {
  activeExperiments = oData.state.activeExperiments;
  
  var mCnt = activeExperiments.length;
  var mTest = '';
  var mVars = '';
  var mExp = '';
  var tCount = 0;

  var noTagArray = ["^hotfix","^preview","^xxx","^notag"]; //Check for notag Experiment names
  var regExMatch = noTagArray.join("|");
  
  for (var i=0;i<(mCnt);i++) {
    mExp = activeExperiments[i];
    var curTest = oData.experiments[mExp].name;
    var curVar = oData.state.variationNamesMap[mExp];

    if (!curTest.match(regExMatch)) { //Eliminate notag Experiments
      if (tCount>0) {
        mTest = mTest + " : ";
        mVars = mVars + " : ";
      }
      mTest = mTest + curTest;
      mVars = mVars + curVar;
      tCount += 1;
    }
  }
  if (mVars!=="") {
    //Data Warehouse Meta Tags        
    mt = $('meta[name="wTag.optVariationName"]'); 
    mt = (mt && mt.length) ? mt : $('<meta name="wTag.optVariationName" />').prependTo('head');
    mt.attr('content', mVars); 
    
    mt = $('meta[name="wTag.optExpName"]'); 
    mt = (mt && mt.length) ? mt : $('<meta name="wTag.optExpName" />').prependTo('head'); 
    mt.attr('content', mTest);       
  }    
}

 

We occasionally bypass the data tagging process to keep results out of our data warehouse. We do this for several reasons but the most common are:

 

  • The experiment is a HotFix between releases
  • The experiment has been turned on for QA, but we aren't ready to track the results

By adding any of the following character strings to the beginning of an experiment name, the experiment can be bypassed by the tagging process:

  • hotfix
  • preview
  • xxx
  • notag

This code also allows for multiple experiments to run simultaneously. If the Optimizely data object contains multiple experiments in the activeExperiments list, the additional names are appended with ":" as a delimiter.

 

Advantages (Strengths)

For us, the biggest advantage to this is the ability to track total event results in addtion to the binary (deduped) results tracked by Optimizely goals. Our sites KPIs are of a nature that require us to examine not just "Did someone engage with a particular site element?" but also "How many times?".

 

It also affords us the ability to track nearly everything the visitor does without generating additional goals specific to an individual experiment. Accessing that data is another story (as mentioned in the footnote) but at least the potential is there for segmentation and analysis.

 

Disadvantages (Pitfalls)

There's no real disadvantage to creating additional ways to track and view data. In our case, we haven't fully developed a scalable method for accessing the data, but having it is a giant step in the right direction. 

 

Other uses of the API

In addition to integrating the experiments with our in house data, we're also using the API for special tracking within Optimizely. I won't go into all the details in this response but some of the uses are:

 

  • Creating custom events (Page View, AJAX calls, Experiment Specific click tracking, etc)
  • Running exclusive, site wide experiments
  • Triggering major page rewrites
  • Connecting experiments between our Desktop and Mobile web sites 

This last item (cross platform testing) may be unnecessary now as Optimizely announced the introduction of a Universal User ID. 

 

For our purposes, we wanted our desktop site to trigger an SMS to a phone. Links to our site from the SMS text would have a query string trigger. On the mobile site, the sibling experiment has an Audience that looks for the query string. Because we want the SMS to trigger a specific variation, we used a custom javascript Audience instead of a simple query string Audience:

 

var me=location.href;
var regExMatch = "sms=true";
if (me.match(regExMatch) !== null){
window['optimizely'].push(["bucketVisitor", 9999999999, 9999999999]);
};
me.match(regExMatch) != null;

 

 

Footnotes

 

*- Data Reporting: Currently, my teams 3 available methods of reporting from our data warehouse are:

 

  • a series of in house reports
  • requesting custom reports from our Business Intelligence team
  • building Excel Pivot Tables from Cube perspectives that are extracted from the main database every morning

This last option is the most used and the most tedious because although we have better control over the form and availability, it is anything but scalable. 

Scott Ehly
Manager of Site Optimization
sehly@rentpath.com

'The single biggest problem with communication is the illusion that it has taken place.' - George Bernard Shaw
stefank 12-02-14
 

Re: What is the most sophisticated integration done with the Optimizely API?

Cool thanks for the response!

 

I think we'll be doing a few similar things, like annotating our own tracking with optimizely experiment data and the like.

 

I'm particularly excited about controling when a test is fired, and to what variant a user gets assigned - that way we can sync up quick UI changes with say our email AB tests. E.g. match a landing page CTA with the one from the email.

 

As for the reporting API, I'll probably end up using that to create a central place to show what experiments are currently running on the site.

 

More reponses appreciated Smiley Wink

Level 2