Rich-text Reply

35% Discrepancy GA vs. Optimizely: due to GA code cookieDomain with leading period? (Drupal related)

andriym 02-16-16

35% Discrepancy GA vs. Optimizely: due to GA code cookieDomain with leading period? (Drupal related)

[ Edited ]

On a recent experiment, I have noticed 35% more "unique visitors" in Optimizely compared to "users" in Google Universal Analytics (segmented for the right dimension to match the right Optimizely experiment, of course).

 

I know that these metrics aren't meant to line up exactly, but I'd think it's reasonable for the two to be within 5%-10% of each other. Or am I wrong?

 

So far, I had a cursory look at some of the docs and something jumped out that could be the reason? Need some advice on whether this is right before I make any changes to my GA code sitewide ...

 

In this Optimizely article: Troubleshooting: Analytics discrepancies, Google Analytics & Universal Analytics , there's a section that claims discrepancies between GA Universal and Optimizely might arise when there's a leading period on setDomainName:

 

Affects: Google Analytics Classic (_gas, _gaq) and Universal Analytics

Does this look like your issue? If not, move on to the next section:

* You see visits only to a limited set of pages on your site.
* Visitor counts between Optimizely and GA/UA don’t match up.

Root cause: You are setting a leading period or www. when calling _setDomainName.
This leads to an additional set of cookies being created. Make sure that if you
have the setDomainName function call on your page it is not using a leading period. For example, use .setDomainName("mydomain.com"); instead of
.setDomainName(".mydomain.com");
[yada yada yada ....] Solution: The easiest way to fix this is to remove the leading period or www. from your
_setDomainName call. If you can’t use that solution, the workaround is to call
_setDomainName before Optimizely sets the customVar, so that the cookies all have
the same domain hash. Depending on how you set up your tracking, you can do that
by adding the following code inside Experiment JavaScript or even in the variation
code...

 

So here's the deal. While my GA code (see below) doesn't use setDomainName as described in that quote above - it does have a cookieDomain instruction that looks very much like the above. This is added site-wide by my CRM's (Drupal) Analytics plugin:

 

<script type="text/javascript">
<!--//--><![CDATA[//><!--
(function(i,s,o,g,r,a,m){i["GoogleAnalyticsObject"]=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,"script","//www.google-analytics.com/analytics.js","ga");
ga("create", "UA-XXXXXX-X", {"cookieDomain":".domain.com"});
window.optimizely = window.optimizely || []; 
window.optimizely.push("activateUniversalAnalytics");ga("send", "pageview");
//--><!]]>
</script>

 

Could really use some advice on this. Is the above bad practice, especially for testing with Optimizely? I typically run URL redirect AB / ABC tests if that helps. Is it advisable to change this up to domain.com without a leading dot? Or even ga("create", "UA-XXXXX-X", "auto"); which is the default? Google docs claim that auto would handle multiple subdomains, which I think is what the CRM module in Drupal is trying to account for with the leading period there. That module is on a "One domain with multiple subdomains" setting currently

 

Any idea how either domain.com or auto would affect my GA data if implemented - especially the tracking of returning visitors vs. new ones? I'm worried if I make the tweak, everyone from that point forward will be viewed as a totally new user even if they had a GA cookie set already ...

 

Thanks!

 

Andriy

Level 3

David_Orr 02-17-16
 

Re: 35% Discrepancy GA vs. Optimizely: due to GA code cookieDomain with leading period? (Drupal rela

Hi Andriy,

 

On a recent experiment, I have noticed 35% more "unique visitors" in Optimizely compared to "users" in Google Universal Analytics (segmented for the right dimension to match the right Optimizely experiment, of course).

 

 

May I ask if the experiment was a completing new experiment or a duplicated version of an old one? If this experiment went through several revisions, I recommend duplicating it and launching the new one. Did you have multiple experiments running at the time? Are you see the discrepancy across all variations?

 

Were you able to verify Optimizely was setting the dimension value on all variations? 

 

 

Could really use some advice on this. Is the above bad practice, especially for testing with Optimizely? I typically run URL redirect AB / ABC tests if that helps. Is it advisable to change this up to domain.com without a leading dot? Or even ga("create", "UA-XXXXX-X", "auto"); which is the default?

 

Most of our users have the domain set to "auto".

 

Any idea how either domain.com or auto would affect my GA data if implemented - especially the tracking of returning visitors vs. new ones? I'm worried if I make the tweak, everyone from that point forward will be viewed as a totally new user even if they had a GA cookie set already ...

 

 

If you would like to maintain your current GA data, I would not advise changing it for the reasons you mentioned. I do recommend reaching out to Google Analytics regarding whether removing the leading period or setting the domain to auto would affect your existing data.

 

David

 

Senior Technical Support Engineer
Optimizely
andriym 02-19-16
 

Re: 35% Discrepancy GA vs. Optimizely: due to GA code cookieDomain with leading period? (Drupal rela

[ Edited ]

David @David_Orr :

 

Thank you for following up on this. I'm posting the answers to your questions - hopefully these can help you shed some light on the issue? Let me know if this opens up any ideas / solutions:

 

* May I ask if the experiment was a completing new experiment or a duplicated version of an old one?

 

I believe I duplicated a previous test BUT edited quite a bit: a new custom dimension slot setup for it in analytics and new target URLs for the variation redirects (this current ABC test redirect URLs are "/" vs "/2h-6" vs "/3h-6" - while the prior ABC experiment was "/" vs "/2h" vs "/4h"); obviously there's also a new unique optimizely experiment ID in this one I'm segmenting for in Analytics when I pull the numbers. So I think on the whole those things make the current experiment "different" enough not to conflate data in Analytics with the old one.

 

* Did you have multiple experiments running at the time?

 

No, we never do. In fact, there was about a month elapsed between the previous experiment and this current one. Only 3% of the Analytics users in the latest experiment also had a custom dimension value from the prior one (but in a different dimension slot).

 

* Are you see the discrepancy across all variations?

 

A great question. I crunched the numbers separately for each variation by comparing its Users metric in Analytics to Unique Visitors in Optimizely - they were all down by the same % as the overall number, so there's no discrepancy there.

* Were you able to verify Optimizely was setting the dimension value on all variations?

 

Yes - here are the results I got from GA Debug:

Original Running command: ga("set", "dimension6", "Optimizely_Exp_########## (##########): Var_########93")
Variation #1 Running command: ga("set", "dimension6", "Optimizely_Exp_########## (##########): Var_########39")
Variation #2 Running command: ga("set", "dimension6", "Optimizely_Exp_########## (##########): Var_########70")

 

Just FYI it fires after ga("create", "UA-XXXXXX-X", {cookieDomain: ".domain.com"}) but before

ga("send", "pageview") - which I think is as it should be. The experiment and variation ids there are consistent with those in Optimizely diagnostic report (and the ones I'm seeing in my Analytics reporting).

 

What do you think? Thanks in advance for your input!

Level 3