Rich-text Reply

Google Analytics Traffic - Too much traffic to one variation

MichaelKoploy 03-25-15

Google Analytics Traffic - Too much traffic to one variation

We just ran an A/B on our site and are having difficulty analyzing the results within Google Analytics. In particular, our "control" saw 14% more unique visitors than the challenger variation. This is something that we have not witness in the numerous other tests we have run over the years.

 

I understand that visitor numbers within Google Analytics and Optimizely should not match up because each defines "unique visitors" differently.

 

We are excluding direct traffic for our site's automated testing tools within both Optimizely and Google Analytics. In addition, the visitor difference is seen across all mediums and all devices (some slight percentage differences between each dimension, but the difference is there).

 

We do use a custom javascript redirect to point visitors to one variation over another.

 

Here is the custom we're using for the redirect:

 

/* _optimizely_redirect=http://custom SA Custom edited code */
var newparam = "layout=var_c0";
var _optly={redir:document.createElement("a")};
var loc = window.location.href.substr(0, window.location.href.indexOf("?") );
_optly.cur=window.location.search;

if(_optly.cur.indexOf('layout=') === -1){
loc += "?" + newparam;
_optly.redir.href=loc;
if (_optly.cur) {_optly.redir.search=_optly.redir.search ? _optly.cur + "&" + _optly.redir.search.slice(1) : _optly.cur;}
window.location.replace(_optly.redir.href);
}

 

The "layout=var_c0" portion is what controls our redirect. layout=var_c0 is our control. layout=var_cf is our challenger.

 

We typically use visitor numbers within Google Analytics to measure conversion rates of individual test variations outside of Optimizely to ensure our tests are safe to go live.

 

Does anyone have any recommendations on where to look to try and understand how & why this is happening?

Re: Google Analytics Traffic - Too much traffic to one variation

[ Edited ]

Hi Michael,

 

Thanks for posting to the Optiverse!

 

Can you confirm the following:

 

  • You've set up the Optimizely-Google Analytics integration on your project and within your experiment
  • The GA dimension set within the experiment matches the dimension you see in your Google Analytics
  • You've added this code (if using Universal GA) to your page's source code BELOW the Optimizely Snippet. (If using Classic Analytics the set up is different)
Example Code
<html>
<head>
<script src="//cdn.optimizely.com/js/XXXXXXX.js"></script> 
</head>
<body>
...
<script type="text/javascript">
// Universal Analytics Setup
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

ga('create', 'UA-40337519-1', 'www.mysite.com');
// Optimizely Universal Analytics Integration code
window.optimizely = window.optimizely || [];
window.optimizely.push(['activateUniversalAnalytics']);

ga('send', 'pageview');
</script>
</body>
</html>

If all that is true, your integration should work correctly and Optimizely results and GA should be very close. GA will take into account the Experiment ID and Variation IDs from all users that enter the experiment.

 

Does that help? Looking forward to hearing back from you and hearing what the community has to say!

 

Best,

 

Amy Herbertson
Customer Success

Re: Google Analytics Traffic - Too much traffic to one variation

Hi Amy,

 

  • Yes, I've set up the integration in our project and within the experiment.
  • Yes, the dimensions should be the same. I am double-checking with our QA and sysadmin to ensure that we are properly excluding our testing tools' IP addresses.
  • Yes, the Optimizely snippet appears right BEFORE the Google Analytics classic (gaq) code.

I created an Advanced segment (using the Key 3 slot) and created segments for two values in our A/B experiment.

 

Attached is a screenshot of the difference in sessions and New Users. I understand these visitor measures will be different between Optimizely and Google Analytics, but this is by far the biggest difference we have had between two variations. This year alone, this is about the fourth major, site-wide test and the only one where we have seen the issue.

 

Screen Shot 2015-04-01 at 11.27.27 AM.png

 

Do you have any other suggestions on how to get to the bottom of this issue?