Seasonal Impact – 7.5% Higher Revenue for Typical Traffic, Opposite Effect During Busy Season
Shipping and tax costs are a critical piece of information that shoppers use to evaluate their online purchases. Shipping/tax estimators are a site feature designed to leverage that information desire. Because of a shopper’s perceived importance of that cost info, sometimes they are willing to offer up something to get it, specifically their e-mail address. Thus, estimators are often hidden behind an e-mail address collection input, but then also placed on pages higher in the conversion funnel (particularly the cart page). The trade-off for the shopper is that they get the important information earlier, but also have to offer up information of their own. The risk for the site is that a shopper may abandon earlier in the funnel due to how they react to the estimator. However, that risk can be mitigated because the e-mail collection can then be used to send out abandonment retention e-mails, perhaps offering a free shipping promotion as enticement to return.
Testing proposal origins can stem from analytics hinting at areas for improvement, evaluation of a new feature or function, or internal battles on which practice works better. In this case there wasn’t a specific problem, our client was simply curious and wished to explore whether the shipping estimator on the cart page had any positive or negative effect on the site.
Screenshot of the cart item grid and shipping estimator:
Shoppers that did not like the costs would bounce whether the cost was shown at the cart page or during the checkout page. Therefore, it is up to the retailer whether they want more shoppers to bounce earlier in the funnel or later in the funnel. Of course, there are other factors that need to be considered, such as if shoppers trying to find the total price would be deterred by the extra step to fill out email address information.
Since the test stemmed from a “discover and see” approach, the prediction was more general. There should not be a significant effect on orders from removing the shipping estimator on the cart page since shoppers that did not like the shipping, taxes, or total price would bounce either way.
We ran 2 experiments, one during low season and one during high season, each with a 1.5 month time period with a huge data sample size. The test during the low season was a 50/50 traffic split. The test during the high season was a 90/10 traffic split to emphasize the results of hiding the estimator where only 10% of the traffic saw the estimator.
The results were surprising, and we ran more tests just to be sure. The slower season trended towards removing the shipping estimator on the cart page. Yet the revenue statistics for the busy season were the opposite; showing the shipping estimator was the winner of the results.
Revenue - Typical Traffic
- Show Estimator - $42.09 per visitor
- Hide Estimator - $45.24 per visitor (+7.5% improvement)
Revenue - Busy Season
- Show Estimator - $79.57 per visitor
- Hide Estimator - $77.53 per visitor (-2.6% loss)
The data led us to investigate how the shipping estimator’s presence affected the shopper’s behavior during different traffic periods. During slow periods the shopper profile is more heavily weighted with repeat shoppers that have a feel of what the shipping and taxes cost. Conversely, during a busier time the shopper profile is weighted more towards first-time shoppers, so showing the estimator may be preferential because they are not as familiar with the site. Without testing, the retailer would not have seen that the seasonality had a pronounced impact on if the shipping estimator should be present.
Some eCommerce sites already display different products and promotions to their various shopper profiles. On a different level, this line of experiments shows that shoppers have distinct habits and their overall behavior can change based on the time of the year. Similar experiments to this one could be run to find out if seasonal features, such as a gift finder or a quick order, may perform better or worse during slow and busy seasons.
Beyond seasonal feature sets, this experiment series opens up a whole other avenue of exploration for future experiments and possibly thousands of new opportunities to personalize the shopping experience based on customer segments. Retailers could test the purchase behaviors of different shopper profiles, how they move through the funnel and what features should show up to optimize conversion. At a micro level, time specific tests could be applied to further determine what should be shown depending if the shopping is done at work, at night, or during the weekend.
This is why cutting edge technology such as Optimizely is vital, so that retailers can quickly test and learn about their shoppers’ behaviors. Based on these results, futuristic eCommerce solutions will allow retailers to rapidly build these dynamic data driven features, an assortment of page layouts, and a variety of page flows so retailers can swiftly transform these ideas into revenue.
At what point was it decided to test between busy season and off season? Did this come up as part of your prediction discussions or do you generally do this as a good practice?
Thanks @MJBeisch, this is a fantastic use case. Personally, I am a big proponent of transparency in the online shopping experience. I want to know exactly what my TOTAL cost is, tax and shipping included. If I can explore ways to save money by shipping slower, I'm much more likely to buy there than on a site where I'm only told tax and shipping after I've submitted my final purchase. Sticker shock is not fun.
I did have a question about your test design. Are you indeed showing the actual tax and shipping cost to the user before they make the final purchase, and are you doing so in both the show and hide scenarios? I presume the answer is yes because you know where and how many users are bouncing.
Another next test could be, if you hide the estimator, to see where to place the reflection of the tax and shipping costs in the funnel to reduce bounces (i.e. does revealing sticker shock earlier in the process give the user time to adjust and accept, or does revealing it later matter less to the user because they are nearly done with the process).
Or, on the funnel step where bounces are the highest, offer a shipping deal/discount. This could be used in a multivariate test in combination with showing/hiding the estimator. You could also use Audiences to offer the discount to frequent shoppers as a loyalty reward and/or to guests/first-time buyers as a welcome reward.
Solutions Architect | Optimizely, Inc.
Interesting that you mention discount/promo offers to certain audiences. This is something that I have tested on other sites. Generally I found that the success of something like that was highly dependent on a site's overall customer profile and marketing strategy. Results varied wildly from site to site.