Rich-text Reply

Optimizely Experiement Data Discrepencies with Internal Data Sources

ADooley 06-01-16

Optimizely Experiement Data Discrepencies with Internal Data Sources

We are using conditional activation to trigger our experiment. Additionally, a custom event is setup to fire for the same conditional activation item. Therefore, we expect the experiment visitor count and custom event number to match - both triggered for a specific page view.

 

Currently, the total custom event count is roughly 33% lower than the total visitor count. At the variation level it ranges between a 27 - 47% delta.

 

We are under the impression that this may have something to do with the time intervals of the custom event firing in sync with our site. Is that possible?

 

Secondly, in attempting to validate more of the data points, I compared the experiment's Thank You Page custom events with our internal data. The Thank You Page event data seems to be roughly 40% lower than our internal data if I apply standard daily averages. (Experiment was applied to 100% of traffic)

 

Does the time interval for data capture for custom events typically varying in this way? Should we assume that all custom event data is reduced by roughly 30-40% or another way to put it, should we assume that Optimizely fires custom events within an experiment on average 60-70% of the time?

 

We need to try to benchmark this data delta somehow before moving forward with any confidence in the testing.

Level 1

AllisonR 06-03-16
 

Re: Optimizely Experiement Data Discrepencies with Internal Data Sources

[ Edited ]

Hi there,

 

Regarding the custom event discrepancy: how are you triggering the custom event? Is it in the Experiment JavaScript? Is there any logic that would prevent the event from being triggered? I wonder if there is a timing issue where, if the event is triggered before the experiment is activated then it won't be attributed to the experiment. I'm happy to take a closer look if you share the experiment id.

 

To address your Thank You page discrepancy: when you're analyzing your backend data, are you de-duplicating conversions that are triggered by the same visitor? Optimizely uses an optimizelyEndUserId cookie to de-duplicate conversions. This means that one visitor can trigger a Thank You pageview multiple times but will only be counted as one conversion on your Results page. Could this explain it? 

 

When you step through the funnel yourself, are you able to see the Thank You page pageview event triggered in the network data? Our support article here outlines how to check for this. 

 

  • If not, and you can verify that the snippet is installed on the page, it may be that the Thank You page is dynamically rendered without a page refresh. See our support article on single page applications for more information about why this is an issue with Optimizely as well as workarounds for this. 
  • If yes, and you can verify the URL matches the targeting for the goal (the URL match validator can be used to check this), then the reason for the discrepancy is most likely due to a portion of your site's visitors blocking tracking scripts like Optimizely. See Kathryn's response to this Optiverse thread for more details about this. 

 

Hope this helps!

 

Best,

Allison
Sr. Technical Support Engineer
Optimizely | EMEA

Allison
Sr. Technical Support Engineer
Optimizely | EMEA
Optimizely