Rich-text Reply

Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

MJBeisch 09-16-14

Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

[ Edited ]

Shipping and tax costs are a critical piece of information that shoppers use to evaluate their online purchases. Shipping/tax estimators are a site feature designed to leverage that information desire. Because of a shopper’s perceived importance of that cost info, sometimes they are willing to offer up something to get it, specifically their e-mail address. Thus, estimators are often hidden behind an e-mail address collection input, but then also placed on pages higher in the conversion funnel (particularly the cart page). The trade-off for the shopper is that they get the important information earlier, but also have to offer up information of their own. The risk for the site is that a shopper may abandon earlier in the funnel due to how they react to the estimator. However, that risk can be mitigated because the e-mail collection can then be used to send out abandonment retention e-mails, perhaps offering a free shipping promotion as enticement to return.

 

Question

Testing proposal origins can stem from analytics hinting at areas for improvement, evaluation of a new feature or function, or internal battles on which practice works better. In this case there wasn’t a specific problem, our client was simply curious and wished to explore whether the shipping estimator on the cart page had any positive or negative effect on the site.

 

Screenshot of the cart item grid and shipping estimator:

 

shipping estimator screenshot.jpeg

 

Hypothesis

 

Shoppers that did not like the costs would bounce whether the cost was shown at the cart page or during the checkout page. Therefore, it is up to the retailer whether they want more shoppers to bounce earlier in the funnel or later in the funnel. Of course, there are other factors that need to be considered, such as if shoppers trying to find the total price would be deterred by the extra step to fill out email address information.

 

Prediction

Since the test stemmed from a “discover and see” approach, the prediction was more general. There should not be a significant effect on orders from removing the shipping estimator on the cart page since shoppers that did not like the shipping, taxes, or total price would bounce either way. 

 

Testing

We ran 2 experiments, one during low season and one during high season, each with a 1.5 month time period with a huge data sample size. The test during the low season was a 50/50 traffic split. The test during the high season was a 90/10 traffic split to emphasize the results of hiding the estimator where only 10% of the traffic saw the estimator.

 

Analysis

The results were surprising, and we ran more tests just to be sure. The slower season trended towards removing the shipping estimator on the cart page. Yet the revenue statistics for the busy season were the opposite; showing the shipping estimator was the winner of the results.

 

Revenue - Typical Traffic

  • Show Estimator - $42.09 per visitor
  • Hide Estimator - $45.24 per visitor (+7.5% improvement)

Revenue - Busy Season

  • Show Estimator - $79.57 per visitor
  • Hide Estimator - $77.53 per visitor (-2.6% loss)

 

The data led us to investigate how the shipping estimator’s presence affected the shopper’s behavior during different traffic periods. During slow periods the shopper profile is more heavily weighted with repeat shoppers that have a feel of what the shipping and taxes cost. Conversely, during a busier time the shopper profile is weighted more towards first-time shoppers, so showing the estimator may be preferential because they are not as familiar with the site. Without testing, the retailer would not have seen that the seasonality had a pronounced impact on if the shipping estimator should be present.

 

Summary

Some eCommerce sites already display different products and promotions to their various shopper profiles. On a different level, this line of experiments shows that shoppers have distinct habits and their overall behavior can change based on the time of the year. Similar experiments to this one could be run to find out if seasonal features, such as a gift finder or a quick order, may perform better or worse during slow and busy seasons.

 

Beyond seasonal feature sets, this experiment series opens up a whole other avenue of exploration for future experiments and possibly thousands of new opportunities to personalize the shopping experience based on customer segments. Retailers could test the purchase behaviors of different shopper profiles, how they move through the funnel and what features should show up to optimize conversion. At a micro level, time specific tests could be applied to further determine what should be shown depending if the shopping is done at work, at night, or during the weekend.

 

This is why cutting edge technology such as Optimizely is vital, so that retailers can quickly test and learn about their shoppers’ behaviors. Based on these results, futuristic eCommerce solutions will allow retailers to rapidly build these dynamic data driven features, an assortment of page layouts, and a variety of page flows so retailers can swiftly transform these ideas into revenue.

Matt Beischel - E-Commerce Optimization Specialist CohereOne

Level 2

Re: Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

Great analysis and summary, Matt. Thanks for posting!
Keith Lovgren

michaelwei 09-17-14
 

Re: Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

Thanks for posting this, it was great to hear the thoughts and insight into the hypothesis and analysis of the results!

At what point was it decided to test between busy season and off season? Did this come up as part of your prediction discussions or do you generally do this as a good practice?
Optimizely

Re: Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

[ Edited ]

Thanks @MJBeisch, this is a fantastic use case. Personally, I am a big proponent of transparency in the online shopping experience. I want to know exactly what my TOTAL cost is, tax and shipping included. If I can explore ways to save money by shipping slower, I'm much more likely to buy there than on a site where I'm only told tax and shipping after I've submitted my final purchase. Sticker shock is not fun.

I did have a question about your test design. Are you indeed showing the actual tax and shipping cost to the user before they make the final purchase, and are you doing so in both the show and hide scenarios? I presume the answer is yes because you know where and how many users are bouncing.

Another next test could be, if you hide the estimator, to see where to place the reflection of the tax and shipping costs in the funnel to reduce bounces (i.e. does revealing sticker shock earlier in the process give the user time to adjust and accept, or does revealing it later matter less to the user because they are nearly done with the process).

Or, on the funnel step where bounces are the highest, offer a shipping deal/discount. This could be used in a multivariate test in combination with showing/hiding the estimator. You could also use Audiences to offer the discount to frequent shoppers as a loyalty reward and/or to guests/first-time buyers as a welcome reward.

Harrison Krat
Solutions Architect | Optimizely, Inc.
harrison@optimizely.com
 
MJBeisch 09-18-14
 

Re: Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

On this site, the tax and shipping costs are always shown on the checkout page, prior to completing the order. There's no hidden fees or "sticker shock" experience to speak of. Checkout is also where a customer makes a selection in regards to their desired shipping method. Only the cart page (the page prior to checkout) was affected by the experiment. The experiment focused on when in the funnel is the best place to present the shipping costs, and if a customer is willing to give up something to get that information earlier.

Interesting that you mention discount/promo offers to certain audiences. This is something that I have tested on other sites. Generally I found that the success of something like that was highly dependent on a site's overall customer profile and marketing strategy. Results varied wildly from site to site.
Matt Beischel - E-Commerce Optimization Specialist CohereOne

Level 2
MJBeisch 09-18-14
 

Re: Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

Oops, I forgot to answer the question @michaelwei asked as well. The decision was made to do seasonal testing after we ran the experiment the first time, which was during an average traffic period. During an analysis meeting/phone call, the client happened to ask aloud "I wonder if we would see the same results in August?" (their busy period). I, of course, responded "well, we should test that!"
Matt Beischel - E-Commerce Optimization Specialist CohereOne

Level 2
MJBeisch 10-16-14
 

Re: Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

Follow-up to this experiment. We are now running the same experiment a 3rd time now that the busy season has passed. It's still too early to call it with confidence, but so far it looks like the shopping behavior that we saw for the typical traffic period is being confirmed.
Matt Beischel - E-Commerce Optimization Specialist CohereOne

Level 2

Re: Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season

Thanks for updating us on your terrfic test, MJBeisch!

Keith Lovgren