Rich-text Reply

Optimizely bugs and data issues - not built for Ecommerce?

wowzadj 04-30-14

Optimizely bugs and data issues - not built for Ecommerce?

[ Edited ]

Two very important activities for merchants are to 1) attribute customer acquisition to a channel 2) encourage repeat purchases by customers driving LTV.  Optimizely fails on both these points.


We have made numerous calls and emails with optimizly support.  We have poured over the raw data and have confirmed the following.


1) For any experiment take a look at Segmentation -> By Source Type -> Referral.  Do you see extremely high conversion rates & you use PayPal Express.  Does that conversion rate align with the % of your sales made with PayPal?  What is happening is Optimizly is attributing all those sales to a PayPal "referral" and not Adwords, Facebook, or the true source of that conversion.


We have been doing extensive testing on landing page optimization for specific channels and this has corrupted our results.


We have proven this issue with Optimizely support however this severe bug has not been addressed.  They just simply need to exclude PayPal urls as the referrer and the original source type will remain.  


2) Optimizely performs "DE-DUPLICATION" on conversions for the same customer during an experiment.  So if the customer appreciates your service and returns to purchase again, that follow-up conversion is not counted toward the test variant.  We have been running tests specifically to drive repeat purchases so clear that was wasted effort.  


First revenue is not deduplicated so your revenue numbers won't align with conversions therefore your revenue/conversion metrics are off.  Second each conversion goal is independently deduplicated.  So if you segmented conversions under $50 and $50+ and you had a customer make a $10 & $100 purchase, the conversion would only be counted once for total conversions but then once each in your segments.


We made multiple phone calls & emails before this "DE-DUPLICATION" was disclosed.  Yet received conflicting information and we still can't get the raw data to match the results page.  "DE-DUPLICATION" was just disclosed by Optimizely in the last couple of days, invalidating many tests we have run in the last 6 months.


If the source of the customer & repeat purchases are important for your test (and your ecommerce business), do no use Optimizely until these issues are resolved.


Dan Siroker, please have the team play less ping pong and focus on these serious defects.

Level 2

MeganBush 04-30-14

Re: Optimizely bugs and data issues - not built for Ecommerce?

Interesting post. Clearly using the native functionality of Optimizely isn't catering to your company's needs as you'd like. Have you linked your tests to different analytics platform to get more data? Because every client is different, we often find that we need different information from different clients. Sometimes the de-duplication of goals works in favor of what we're trying to acheive, and sometimes it doesn't. But with minimal effort, we link every test to a corresponding analytics account, where we have a "second set of eyes" with which to compare the Optimizely results.


As far as the Paypal referral issue, can you set up a custom segment which separates Paypal referrals from other referrals?


Admittedly one of the challenges I've had with my experience in working with Optimizely is that clients think it can do anything and everything out of the box. While I truly believe it can do just about anything, I've learned sometimes it takes a bit more effort to get exactly what I need from it as my needs become more specific to a particular client or test.

Megan Bush
wowzadj 05-01-14

Re: Optimizely bugs and data issues - not built for Ecommerce?

[ Edited ]

Megan, these are not an edge case.  We sell with PayPal.  We want repeat customers.  This is normal for ecommerce sites.  If you are unfamilar with ecommerce then please connect me with a Product Manager who understands the importance of customer aquisition atribution and customer life time value.


We've invested several hours investigating these issues, and providing data to Optimizely support.  There hasn't been any movement.  We are a Gold customer so this is disappointing.


PayPal Bug (ticket 42675)

We've gone over the raw data with your support team.  They agree it is a bug.  We need you to fix it.Here is the current flow flow bug:



Google (referer) -> our site -> our cart -> paypal checkout (new referer) -> our thank you page



Google (referer) -> our site -> our cart -> paypal checkout (URL ignored as referer) -> our thank you page


You need to exclude PayPal URLs as a referer source and that will preserve the original true referer.  Then we will properly see how the test is effecting our other segments such as Adwords. 



Deduplication Explict Option (ticket 47565)

This needs to be an explicit option that can be set when starting a test.


Repeat customers are everything in retail.  We've run tests specifically trying to increase repeats, however Optimizly was secretly discarding this data.  


It's difficult to overstate how much this underminds ecommerce testing.  For us this discovery of excluding our repeat customers in a test will invalidate 20+ tests from the past 6 months.  

Level 2
MeganBush 05-01-14

Re: Optimizely bugs and data issues - not built for Ecommerce?

I am actually very familiar with ecommerce, as I work exclusively with ecommerce clients daily. I don't work for Optimizely, so I can't speak to your communication with Optimizely support. I just saw your post and thought I could help by providing some alternative options for your team until your feature requests are addressed/implemented.

Megan Bush
wowzadj 05-01-14

Re: Optimizely bugs and data issues - not built for Ecommerce?

Thank you Megan.  I mistook your reply for being an official response.  Thank you for making suggestions.  Having to toss out months of tests and putting a halt to our testing has cause my frustration to boil over a little.


Does Optimizely monitor this board?


To answer your question we have also tried VWO, GA, and a homegrown solution.  We found content experiments to be the worst of the lot and the hardest to configure.  We went for Optimizely over VWO due to the server side API.  We test business logic in addition to visual changes.  However once we realized there was an issue we've been running pure Optimizely code to rule out and external factors.


As an ecommerce professional I'm sure you agree having complete & accurate conversion data is paramount in testing.  

Level 2
Greg_Tate 05-01-14

Re: Optimizely bugs and data issues - not built for Ecommerce?


One of the main goals of Optiverse is to shorten the feedback loop between Optimizely and our customers so we’re really grateful for this feedback.


Optimizely is built to work for the majority of e-commerce and online retail sites. However, some customers have specific needs that go beyond what the tool provides out-of-the-box. In those situations, we provide custom events, custom dimensions, and analytics integrations that allow customization.


Currently, Optimizely looks at the most recent source of traffic to determine the source for a visitor, not the original source for that session. This can be resolved by integrating Optimizely with a third party analytics platform (such as Google Analytics) which may align better with your business needs.


By integrating with GA or the analytics platform of your choice you should be able to tap into any metrics/goals which you have already set up in that system. The Optimizely integration will allow you to set and adjust those metrics based on Optimizely experiment/variation. You can read more about integration options in the following KB article:


We have revised our documentation to highlight that the Revenue goal is the only aggregating goal, and all other conversion goals within Optimizely will de-duplicate to unique conversions by default.


We have also updated our Default Dimensions documentation to alleviate any confusion about how source/referral tracking is done. Learn more in the following KB article:


Thanks again for the feedback and patience. We are committed to iterating on our documentation to make these nuances clear, and are continually working to improve our product!


Greg Tate


PS - If you'd like to continue using Optimizely results, there is a workaround to disable de-duplication of goals. By sending us anonymous tracking calls with the user removed, you can get non-Revenue conversions to show up in the results without deduplication.

wowzadj 05-02-14

Re: Optimizely bugs and data issues - not built for Ecommerce?

Thank you for the reply Greg.  Again, this is not an edge case.


A customer paying with PayPal is never a channel for any merchant.  Referer != PayPal.  The fix is easier than becoming session based.  When setting the referer, simply have a conditional if the HTTP referer is a PayPal URL, then don't update the visitor's referer.  One line of code and your feature will work as expected by every merchant, and we will have access to to accurate realtime information (unlike GA).  The feature you are selling.


On deduplication, why the resistance to making this a configuration option when creating a new experiment.  The footnote in your docs is better than the no disclosure previously, but it can get better.

Level 2

Re: Optimizely bugs and data issues - not built for Ecommerce?

I'm curious, what analytics tools are you using on your site and do they behave in the way that you want Optimizely to behave? I'm not being snarky, I'm genuinely curious to see if there's something that can smoothly handle the cookie chain on a 3rd-party domain, without having code for the tool on the 3rd-party domain.


Having worked with Google Analytics a lot, I know that 3rd-party domains can cause issues for all kinds of tracking, whether it's a PayPal payment page or getting sent to for OAuth. I would think that GA would give you the same referral source problem, or have you found a workaround? Or are you using a different analytics package?

wowzadj 05-06-14

Re: Optimizely bugs and data issues - not built for Ecommerce?

You do not need code on a 3rd party site.  That third party site is the referer of your testing site where you do have code and that information is part of inbound HTTP request.   Install Live HTTP Headers plugin for your browser and take a look.  Any developer would acknowledge that the fix is extremly trival to implement.


GA has multiple atribution models and fitlers that can be utilized.  However GA is free courtsey of Google.  Not a paid service.


In my experiance it is exceptionally unusual for a vendor to refuse to acknowledge an issue once demonstrated.  Especially on a core feature.

Level 2
db-_-db 05-16-14

Re: Optimizely bugs and data issues - not built for Ecommerce?


Simple workaround:

in your experiment script put code that does the following:

1. if you don't have a "orig_referer" cookie, set it as a session (1 hour cookie)
2. on your confirmation page, use this cookie value as a custom dimension of "Original Referer" which you can create
3. In reports you can break out results by the dimension, i think, so you can check which "Original Referer" can be attributed to your revenue/conversions.

For example

if ($.cookie("orig_referer")) return;
var date = new Date();
var minutes = 60;
date.setTime(date.getTime() + (minutes * 60 * 1000));
$.cookie("orig_referer", document.referrer, { expires: date });

then use a dimension or segment the visitor by reading that cookie.

Hope this helps!

Level 2