Rich-text Reply

Testing on the receipt page - what exactly does revenue goal measure here?

Sergei_Kogut 04-06-15

Testing on the receipt page - what exactly does revenue goal measure here?

We have a test running on the receipt page, which is a thank you page after a customer completed a paid order. It's also usually the last page they view, it's our exit page. 

 

So:

Our experiment is tageted only to the receipt page.

That means this test's visitors are counted when they visit this page and nowhere else.

And we have Optimizely's revenue reporting tag firing on that same page.

So what exactly is it measuring?

 

A) The current order value that each of these visitors have placed just on the previous page?

B) Or the future revenue, just from their subsequent visits?  

C) Both A and B above?

 

If it's A), this kind of makes it difficult to measure any real impact of changes made to the Receipt page.

 

If it's only B), then why is revenue showing up already, after just a few days of this test running? Our customers don't usually come back so soon, maybe after a month. 

 

If it's C) both, curent and future revenue, then it's again hard to separate any actual impact we're making on the future revenue, from the random noise generated by the whide range of AOVs of the current orders as they come in. 


What's your advice on this? How do you measure tests on your receipt pages? 

VP of Marketing at FSAstore.com
Amanda 04-06-15
 

Re: Testing on the receipt page - what exactly does revenue goal measure here?

Great question. I will let the Optiverse answer from a strategic goal creation point of view, but from a technical perspective the revenue goal is probably firing as soon as someone lands on the confirmation page (which could be the first time they land in the experiment.) On your thank you page, you have code that looks like this: 

 

<script>
window.optimizely = window.optimizely || []; window.optimizely.push(['trackEvent', 'eventName', {'revenue': valueInCents}]);
</script>

 

The revenue goal tracks whatever you have populated in the valueInCents, which I supsect is the value from the purchase on the page immediately prior. You can confirm this if you want by looking at the network traffic when hitting this page in a test purchase to see what value gets passed. You'll want to look at the "v=" parameter for amount.  

 

Since it sounds like your experiment is trying to influence repeat purchases, there may be other goals you could be tracking instead. Maybe the Optiverse has some ideas here?  Perhaps there is a way to write some logic into the experiment to ensure that the revenue code only fires on repeat purchase. I'll noodle on this a little, and wait to see if any other great ideas pop up here. 

Optimizely
Hudson 04-07-15
 

Re: Testing on the receipt page - what exactly does revenue goal measure here?

@FSA_Store

 

If I'm understanding correctly, you'd like to test whether an alternate experience on your receipt page will influence future purchase conversion. Cool test!

 

The revenue goal should work as you'd like it to, but it sounds like your implementation might need to be tweaked per Amanda's comment, such that it only captures any conversion event after that receipt page experience is loaded. 

 

You could also proxy revenue via future purchase clicks (click or custom event), which directionally should answer your question - any further purchase clicks should correspond to 

 

Note that this test may take quite a long time to reach fruition, depending on your existing baseline of returning visitor % as a basis of overall site traffic, and standard purchase conversion rates - it may be helpful to add some 'higher funnel' goals, like product detail page views, add to carts, etc., so that you can observe directional improvements between metrics that should all have similar improvements if your test's hypothesis proves to be true!

 

Does this help? Let us know how this tracks to your testing goals. 

 

Happy Testing,

Hudson

Optimizely

Re: Testing on the receipt page - what exactly does revenue goal measure here?

@FSA_Store

 

Regarding your question about strategic goal creation, I have a couple recommendations for additional metrics that you could use to measure the success of a test like this.

 

Since your test is targeted to your purchase confirmation page, I will assume (as Amanda did) that you are looking for return visitors and incremental transactions upon completion of the current transaction.

 

Revenue Per Visitor (Total Revenue) will be a key metric – you have this covered, and Amanda seems to have answered your question regarding the calculation and functionality of this goal.

 

In addition to that, you could try the following if they make sense for your unique experiment.

 

(1) Associate sales to a unique user.

 

One idea would be to take some of the ideas from the beta feature, Universal User ID. See more here: http://developers.optimizely.com/javascript/#universal-user-id-(beta)

 

In this case, the user's email could be captured from any login/signup form and then set as the Univ. User ID. However, since this is personally identifiable information, it will need to be hashed. A developer can add a basic hashing method to the String prototype of JavaScript by using the following code: 

 

String.prototype.hashCode = function() {

  var hash = 0, i, chr, len;

  if (this.length == 0) return hash;

  for (i = 0, len = this.length; i < len; i++) {

    chr   = this.charCodeAt(i);

    hash  = ((hash << 5) - hash) + chr;

    hash |= 0; // Convert to 32bit integer

  }

  return hash;

};

 

So, for example, you could say:

 

    "usersemail@gmail.com"hashCode();  //=> generates random UNIQUE hash for this string that can be used as some identifying factor

And this would generate a unique hash for the email. This could then be sent as the Universal User ID. This approach is really just a way to take an identifiable piece of information for a customer (the email) and attach it to the order, both current order and any future orders. This could give meaning to your data for further analysis to determine how much user X with hashed Universal User ID “38992871” makes on all orders to see if there's in increase/decrease/no change at all. 

 

You could use the Optimizely JavaScript API to push a custom event with this ID or unique identifying information. You could also push a custom event with the order number specifically. This could be added to the code where the revenue tracking is occurring. More info here: 

 

https://help.optimizely.com/hc/en-us/articles/200039865-Revenue-tracking-goals#best_practices

 

I would suggest looking at: Best Practices, Unique revenue by order (deduplication), Revenue per product, and Remove outliers.

These sections contain some additional information that could be used to help solve this problem.

 

Custom event tracking can be found here: https://help.optimizely.com/hc/en-us/articles/200039925-Custom-event-goals

 

A custom goal would have to be created and added to the experiment, and then the JavaScript API call would push that event with a specific value (in this case, an order # for example)

 

This of course would require a developer since it requires custom JavaScript code - and yes, one of our star developers did assist with this particular description.  Thanks @seanemmel_ba!

 

 

(2) Measure Time on Page for the Confirmation Page

 

If the confirmation page has new messaging that you want your customers to really digest, I would want to know how much time they spent on that page. You could easily do this if you integrated the test with your web analytics.

 

I wanted to include a screenshot to show you how you could view a specific test’s Avg Time on Page, but I'll describe it for now.

 

Find your custom variable in analytics (if you use GA) then add "Page" as your secondary dimension and apply an advanced filter to view your confirmation page with the test's Original & Variation.  Finally, you will have to customize your metrics to pull "Avg Time on Page" into the dashboard.

 

If time on page is greater for your Variation, chances are they are dwelling longer to consume the new messaging on the page.

 

 

Hope this helps!