Rich-text Reply

Surprising Result: Unintended Consequences

MJBeisch 07-29-14

Surprising Result: Unintended Consequences

[ Edited ]

Split testing on e-commerce sites is definitely important for increasing revenue and order conversion. However, testing can also be an important form of protection against doing damage to your current state. Here's one such example.



About a year ago, one of our clients migrated to a newer, more modern platform for their e-commerce site. Overall the process was pretty smooth and the new platform offered more robust features and a much quicker code-to-deployment turnaround time. Post migration, this client was seeing a higher volume of orders, but a decreased AOV, and thus overall less total revenue. The design of the site had not significantly changed during migration. In fact, a lot of care was taken to try and match platform implementations as closely as possible so that site continuity was maintained. Thus, the question for the experiment: "Why are orders up, but revenue is down?"



After some careful analysis, we discovered one small but potentially significant user page flow change that had been made. The new site had added the capability for the site to have a "floating cart". This feature is a shopping cart that is persistent on all site pages and accessed via hover state in the header. Here is an example of it in action:




Note the "Checkout" button at the bottom of the floating cart. This button allowed a user to skip the cart page entirely and proceed directly into the checkout process. On the old site, there was no floating cart and the user was forced through the cart page when making a purchase. The cart page was merchandised with targeted cross-sells and sale items. This divergence in page flow experience formulated the hypothesis for this experiment: "Shoppers are skiping directly to the checkout page via the floating cart. They are not exposed to the Special Offers & Cross-Sells that are on the cart page, causing missed opportunities to add additional products to the cart."



A solution for this seemed rather apparent: force everyone through the cart again. However, we still wanted to test and be absolutely sure. The predicted result of the experiment was: "Forcing all shoppers through the cart page during the checkout process will increase revenue by increasing items-per-cart."



The experiment itself was rather straightforward, a A/B split test. Variation A was the baseline: 50% of traffic experiences current behavior. Version B was the comparison: the other 50% of traffic is forced through the cart page by removing the "Checkout" button.






The experiment ran for a little over 2 months during an average (typical) traffic period for the site. Based on information collected from Optimizely, Google Analytics and internal platform data, it was determined that version B resulted in a higher AOV. However, revenue for Version B was down 1.2% versus baseline. How are we seeing a higher AOV but lower revenue? We took a look at pageview conversion rates for the conversion funnel pages being tested: cart, checkout, and summary.


Cart Page

  • Version A - 7.05% conversion rate
  • Version B - 9.66% conversion rate (+36.9% improvement)

Checkout Page

  • Version A - 8.38% conversion rate
  • Version B - 7.36% conversion rate (-12.2% improvement)

Summary Page

  • Version A - 5.54% conversion rate
  • Version B - 5.32% conversion rate (-4.0% improvement)

For Version B, higher conversion was expected on the cart page since the variation forced users through that page. However, lower checkout and summary page conversions was unexpected! Version B was causing higher cart abandonment between the cart and checkout + summary pages.


These results allowed us to infer that the hypothesis was indeed correct; users are missing out on the offers and cross-sells presented on the cart page, as demonstrated by the higher AOV on Version B. However, the predicted solution was not the correct one; users preferred the new page flow and abandoned more when forced through the cart, as demonstrated by the cart & checkout pageviews.



If our client had proceeded with implementing their proposed solution without testing, they would have ended up causing even more revenue loss. Instead, further experiments eventually allowed us to find a solution that both increased order size while maintaining order conversion rates, leading to an increase in revenue and solving their AOV migration issue.

Matt Beischel - E-Commerce Optimization Specialist CohereOne

Level 2

Re: Surprising Result: Unintended Consequences

That's a great documentation of your test. Kudos, Matt!
Keith Lovgren

Re: Surprising Result: Unintended Consequences

Wow! Thank you for sharing this. Invaluable info here. Congrats on figuring that out and implementing a killer test.