Rich-text Reply

Do you use both Classic and Optimizely X? Do your results match up?

emgao-fsa 03-06-17

Do you use both Classic and Optimizely X? Do your results match up?

Anyone here use Classic and Optimizely X simultaneously? We typically use Classic for experiments and X for personalization campaigns, but have been suspicious that our Optimizely X platform does not track revenue correctly. Sometimes there's revenue in the hundreds when we expect revenue in the tens of thousands, and other times revenue is just consistently 0! The discrepancies are so large that it definitely could not be from the slightly different times Classic vs X take to refresh the results data. We've filed multiple support tickets over the past few months, and Optimizely has confirmed that our revenue implementation is set up correctly, so there should be no reason why we are missing revenue...

 

I'm curious if anyone else has this issue? If so, what do you do? We want to continue using Optimizely, but if the results reports are consistently unreliable, the platform is basically unusable if we want to be a data-driven company.

 

If you use both Classic and Optimizely X, a fun way to see whether your results are off is to start a Classic experiment and Optimizely X experiment targeted to the same audience, sames pages, and scheduled to start at the same time. I did this by creating "empty" experiments with no actual code, just to see how my page visits, goals, and revenue got tracked for each platform. This will confirm whether your Optimizely X misses revenue and other goals, too. I'd love to know if this is a platform-wide issue or just some weird bug with my account.

robertchan 03-07-17
 

Re: Do you use both Classic and Optimizely X? Do your results match up?

I did something similar to what you wrote in the last paragraph and saw no discrepancies. I'm assuming this might just be a bug. I'd suggest using @ to directly alert an Optimizely employee here on the forum.
Robert Chan

Experimentation Hero
emgao-fsa 03-08-17
 

Re: Do you use both Classic and Optimizely X? Do your results match up?

Hi @robertchan, thank you for testing it out, appreciate it! Can you let me know what type of audience targeting you used? I'm wondering if you did something more general like Everybody or only Desktop, or something more specific. We use quite granular targeting that results in discrepancies between Classic and X, so knowing more about your targeting would be helpful.
robertchan 03-08-17
 

Re: Do you use both Classic and Optimizely X? Do your results match up?

Anytime! Did everybody so maybe your granular scope made the difference.
Robert Chan

Experimentation Hero
emgao-fsa 03-09-17
 

Re: Do you use both Classic and Optimizely X? Do your results match up?

[ Edited ]

Following up on this for anyone who might encounter this issue - Optimizely X evaluates audience targeting on every page load instead of on an experiment level - so query parameter based targeting on its own is pretty much unusable. This is not clearly laid out in their articles about what's new in Optimizely X, so I'll copy and paste Optimizely Support's explanation here:

 

Classic Optimizely evaluates audience conditions at the experiment level, so if a visitor passes audience conditions (query parameters) then they remain in that experiment indefinitely. It is based on their endUserId cookie, so even if they no longer qualify for the experiment they will continue to be included.

 

In Optimizely X, the visitor is evaluated at the page level. So that every time they reach the page they are evaluated and a decision is made if they qualify. So first time when a visitor has the required query parameter, they will see the experiment. If they come back without the parameter, they are evaluated as not passing the audience condition and will not be bucketed into the experiment.

 

In my case, I had targeted my experiment to substring match on my whole site. Visitors enter with a query parameter, but once they go to any other page, they are no longer in the experiment. This also means that if you change the URL Targeting to just the page with the query parameter, if the visitor goes back to that page at any time during their session, they will get evaluated for audience conditions and not pass, dropping them from the experiment.