Rich-text Reply

Experiment Goals don't match data in my CMS

HappyPanda 07-21-14

Experiment Goals don't match data in my CMS

I am using Optimizely to optimise an Ecomm store on Big Commerce.

 

I have a pageview goal set-up for the completed order page. This is to track "Completed Purchases" as a goal.

 

The data is coming through, however the information reported in Optimizely is different to the number of orders I'm seeing in the Big Commerce CMS.

 

Optimizely is reporting less orders than are actually coming through.

 

I'm tracking Pageviews on the "finishorder.php" page with a simple match.

 

Any ideas why there is a discrepancy?

adzeds 07-21-14
 

Re: Experiment Goals don't match data in my CMS

Just so we can try and narrow it down a bit, how much out are we talking?

Is it 1-2 orders of 100s?

My initial thought is that Optimizely is not bucketing all your visitors due to your targeting conditions so it wont track those sales.

Once we know how many orders are affected we can try and narrow down the issue.
David Shaw
Level 11
HappyPanda 07-21-14
 

Re: Experiment Goals don't match data in my CMS

It changes.

 

From a look:

 

Sunday 20th: Optimizely tracked 0 conversions, CMS reported 4.

Saturday 19th: Optimizely tracked 4 conversions, CMS reported 8

Friday 18th: Optimizely tracked 3 conversions, CMS reported 4

Thursday 17th: Optimizely tracked 2 conversions, CMS reported 4

Wednesday 16th: Optimizely tracked 2 conversions, CMS reported 4

 

Admittedly these are small discrepancies, but still they are discrepancies and it's annoying. Could this be an issue with PayPal order completions and the way the user is passed back from PayPal to the confirmation page?

 

 

HappyPanda 07-21-14
 

Re: Experiment Goals don't match data in my CMS

I believe I may have found an explanation.

This is a multi page test, with home, product, category and checkout page mods.

The targeting for the category pages is using simple match targeting for various categories. It covers a large range of the top traffic categories, however not all of them.

If traffic is entering by some of these less popular pages then going on to convert I imagine it's never triggering the Optimizely test and subsequently not being counted for the purposes of the test.

Sound reasonable?
adzeds 07-21-14
 

Re: Experiment Goals don't match data in my CMS

Yup. That was my initial thought in my earlier reply.

"My initial thought is that Optimizely is not bucketing all your visitors due to your targeting conditions so it wont track those sales."
David Shaw
Level 11
adzeds 07-21-14
 

Re: Experiment Goals don't match data in my CMS

If you have your Optimizely test linked with Google Analytics you should be able to confirm this theory?
David Shaw
Level 11
Kathryn 07-23-14
 

Re: Experiment Goals don't match data in my CMS

This sounds like the most likely explanation. If certain pages are excluded from your targeting conditions, then sales originating from these pages will not be tracked by Optimizely.

David's suggestion to compare the data in Google Analytics is a great one and should help you clarify if this is indeed the case. Should you notice anything unusual, of course feel free to come back here and discuss.

-Kathryn
Optimizely
HappyPanda 07-30-14
 

Re: Experiment Goals don't match data in my CMS

I've done a bit more digging into this, and it appears this is not the issue.

 

I am seeing however that only payments made via credit card are coming through into Optimizely. Payments processed via PayPal don't get recorded as a conversion.

 

I'll be trying to fix this today, so will follow up with my solution.

Kathryn 08-08-14
 

Re: Experiment Goals don't match data in my CMS

That's interesting. Is it possible that visitors going via Paypal are not returned to your site after confirming the payment, thus resulting in the log calls not being correctly fired to the Optimizely servers? Otherwise, might it be that the redirect URL reached by confirming an order via Paypal falls outside of your targeting conditions. 

 

I'd be interested to hear what you found!

Optimizely
adzeds 08-08-14
 

Re: Experiment Goals don't match data in my CMS

I would imagine that Kathryn has hit on the right track there with the Paypal visitors not returning to your site.

Don't they have to click a link to return to the merchants site in a lot of cases, or wait 10 secs to be redirected? It may be that they are closing the browser before heading to your success page.
David Shaw
Level 11