Rich-text Reply
Highlighted

Ask the Expert your eCommerce questions!

Amanda 06-30-14

Ask the Expert your eCommerce questions!

[ Edited ]

The live portion of Ask the Expert is now closed. Please submit your question on the apporpriate discussion forum so our other community members can help you out.

 

Respond to this post with questions you may have, bounce ideas around, and get to know the expert!

 

Current Theme:

Strategy for E-Commerce web and mobile optimization. 

 

Our Expert:

Khattaab Khan, a Strategic Optimization Consultant at Optimizely.

 

Who am I?

I support Optimizely’s Platinum enterprise E-Commerce clients through test ideation, results review, UX education, and testing program management. I hope to equip you all with impactful test ideas to drive your business goals as well as recommendations for building an optimization culture.

 

Khattaab has also authored or helped host the following: 

 

 

And what do I do for fun?

 

I’m an avid traveler and student of languages. When I’m not snapping photos and collecting stamps in my passport, I enjoy exploring farmers’ markets to collect the freshest ingredients for my favorite creative outlet: cooking.

 

Ask Away!

 

You can ask questions until 5pm PST on Wednesday, July 2 at which point this session will be closed. Check out the post here for guidelines and additional information.

 

 

 

--

If you are interested in being featured as an Expert for a specific topic, please email optiverse@optimizely.com

 

Also, if you have any ideas for future Ask the Expert Themes, please post them in the Community Ideas board here.

--

Optimizely

Re: Ask the Expert your eCommerce questions!

[ Edited ]

Love this new feature, "Ask the Expert" !

Hi @Khattaab ,

My question relates to your experience with ecommerce experiments.

In your experience, for larger clients who haven't done much testing, is there a general test on a product detail pages you've seen perform well across multiple verticals?

Thanks in advance!

Keith Lovgren

Khattaab 06-30-14
 

Re: Ask the Expert your eCommerce questions!

Thank you for your question @keith_lovgren. Tests to get you started with optimization on your product detail page include:

 

1) Value proppositions adjacent to primary CTA

If you have a key value prop such as "Free Shipping" or "Easy Returns", feature it as close to the primary CTA as possible. In the event that such a value prop as already featured elsewhere (e.g. header), it's acceptable to repeat key value props on the page.

 

2) Prominent breadcrumb

Keep users focused on the product category that they are currently browsing. Users with a less focused path through a site are less likely to purchase.

 

3) Add to cart confirmation

Reveal a window that shows the product in the cart and/or feature a dedicated CTA to continue to checkout.

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5

Re: Ask the Expert your eCommerce questions!

Thanks, Khattaab. 

 

Great tips and I look forward to putting them to good use!

Keith Lovgren

zemaniac 07-01-14
 

Re: Ask the Expert your eCommerce questions!

[ Edited ]

Hi @Khattaab ,

 

Small question:

What would be your suggestion for a small company (12-15 people) with no testing/optimization culture that has just started using CRO tools? What's the best workflow and division of roles from your experience?

 

Thank you!

Alessio

Alessio Romito
Project Manager at 21DIAMONDS GmbH
Level 2

Re: Ask the Expert your eCommerce questions!

Hi @Khattaab ,

 

Thanks so much for taking the time to field our questions! Smiley Happy I have two that I'd love your expert feedback on:

 

First, when prioritizing test ideas, our team considers the importance of the page(s) that will be impacted. One thing we struggle with in regard to that discussion is whether tests on the homepage, for example, are more or less important than tests in the checkout funnel because it receives so much more traffic and, if designed well, could capture the attention of new visitors. What are your thoughts on this debate? In terms of importance, should a page’s volume of traffic or proximity to the checkout funnel win out?

 

Second, when the objective is to test a completely revamped page, is it best to do so in an iterative process with one element changing at a time or all at once with the redesigned page as the one variation? We obviously want to know which changes provide a positive/negative impact, but we also want to know how those elements work together as a whole. What are your thoughts?

 

Looking forward to hearing from you!

 

Laura Hunter

Khattaab 07-01-14
 

Re: Ask the Expert your eCommerce questions!

[ Edited ]

Great question, @zemaniac . Implementing a repeatable process is essential to cultivating an effective testing culture. 

 

An ideal workflow encompasses the following steps:

 

Brainstorm > Prioritization & Roadmap > Variation Design & Documentaton > Test Setup (Goals & Targeting) > QA > Run > Results Analysis & Sharing

 

When managing personnel resources, it's important to have a point person (Testing Program Manager) that will either be personally involved in the ideation, implemenation, QAing, and results analysis/sharing processes for each experiment or have a high level of visibility over what is happening in each Optimizely project.

 

Harnessing the appropriate techical resources early on is also essential. Even if the scope of the first phase of testing does not necessitate the involvement of internal developers and designers, it's important to keep these teams informed of what is happening in the Optimization program through effective test documentation and results sharing.

 

Results sharing feeds the iterative testing loop and builds confidence in the power of testing and personalization among those stakeholders who are not touching Optimizely everyday so that as testing becomes more sophisticated and involves more business unitis, a business case can be made to dedicate more resources to it.

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5
Khattaab 07-01-14
 

Re: Ask the Expert your eCommerce questions!

Thank you for your questions @Laura_Hunter. The answer to your first question about prioritizing tests based on traffic volume vs. proximity to the primary conversion event depends on the level of traffic you have on your product and checkout pages. If this volume is very low, it will take a long time to achieve results that have sufficient statistical significance and statistical power (see the Knowledge Base post on How long to run a test for additional context).

 

While having a good mix of tests on different pages provides good insight to how changes along the path influence behavior down-funnel, focusing on those tests that are closest to your primary conversion event are most valuable because the goals assigned to such tests most likely have a more direct inpact on your key success metrics, thereby enbling you to more accurately communicate how variaiton changes create business value. You are serving experiments to a more qualified audience: users who have already communicated a high level of interest in your offering and have spent a good amount of time on your site.

 

Regarding testing revamped designs, an iterative process provides more insight than a page redirect because, as you correctly identified, a redirect will not tell you what about the redesign influenced a change in behavior. To streamline the tedious process of testing element variations one-by-one, consider grouping related elements in a multivariate test (MVT). The elements that you group should all have the same goal on the page (i.e. drive CTA click, encourage form complete, etc.). An MVT will run all possible combinations of variations to provide very specific insight into how elements do or don't complement one another.

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5

Re: Ask the Expert your eCommerce questions!

Excellent! Thanks so much for your feedback, Khattaab! I appreciate it!

Laura Hunter
nolanmargo 07-02-14
 

Re: Ask the Expert your eCommerce questions!

[ Edited ]

Good Morning @Khattaab ,

 

We're currently in the process of making our sales funnel responsive.  My questions have more to do with UX best practices on mobile rather than testing:

 

  1. What are some best practices for handling interaction points, for example, when a user is requested to sign-in to purchase OR save an item in their cart?   Currently, on our full site, we use modals for these interactions, but we're concerned that this method might degrade the user experience, not to mention the difficulty of implementation. Any examples or suggestions would be greatly appreciated.
  2. With regards to forms in responsive, is it best to stretch these across a few pages, the thought being smaller bites are more digestible, OR should we should we not fear a long scrolling form?
  3. Generally speaking our creative team likes to add a lot of art, which lends itself well to the fullsite.  In your experience with mobile, do superflous images & creative treatments ADD or DETRACT from the responsive sales funnel experience on mobile?  Is simplicity key OR have you observed creative treatments enhancing throughput and UX?

Any and all thoughts and best practices are welcome - feel free to reach out if any of my questions are unclear!

 

Happy Testing!

 

Nolan

 

JohnH 07-02-14
 

Re: Ask the Expert your eCommerce questions!

Hi @Khattaab , we are thinking about adding a pop-up on our site to collect email addresses. Can you provide some best practices on how to determine where in the user flow is the best time/place to interrupt a user with a pop-up? Also, if you can provide any examples of how other customers have successfully implemented an email collection pop-up that would be very helpful. 

Level 2
Amanda 07-02-14
 

Re: Ask the Expert your eCommerce questions!

Hey Khattaab - our member @rcarrollinogen also had another question related to pop-up forms. Can you provide him with some best practices? His question is here

Optimizely
nolanmargo 07-02-14
 

Re: Ask the Expert your eCommerce questions!

[ Edited ]

John - we've recently run a series of lead capture tests - using a modal lightbox technique. We identified a candidate page that receives thousands of visitors daily, through an inherited SEO tactic, that represented roughly 45% of our daily visitor volume. Despite the high number of visits, the sessions were purely transitory, at times a >97% bounce rate, all while providing ZERO economic value.

We quickly drafted up 6 variations, and called them on the highly trafficked page and were surprising to see each day our email address list pile & pile up. We now have a steady-stream of 'leads' pulling through. Even without optimizing our email series, this new cohort of 'submitters' is now 5x more likely to head down our conversion funnel and discovery more about our product.

Since the initial test, we ran a couple more variations, one that 'delayed' the modal display by 10 seconds, which flopped and another, which removed a restrictive opt-in statement that actually continued to work as well as the original (this is the one we are currently running).

As Khattaab will probably back me up, be wary of offer testing (i.e. discounting) for those, who come from less qualified sources OR like in our case, exhibit user behaviors that may not lead to conversion, even WITH an offer ... search out your 'qualified' visitors and target them with your discounts as incentive for an email submission Smiley Wink

Hope that helps - PM if you'd like some examples &OR techniques.

Khattaab 07-03-14
 

Re: Ask the Expert your eCommerce questions!

@nolanmargo , please consider the following in context of UX best practices on mobile:

 

1. Modals layered on top of content on a small mobile screen is not a the best experience. Instead, direct users to a page that has only one task: registraion. An example of this flow when an unregistered user adds an item to their cart is the Gilt mobile website.

 

2. Input fields on mobile work well when they are broken-up, thereby creating a sense of achievement through a process. Complementing this format with a progress bar across the top/bottom of the screen provides users proper context of where they are in the process and may compel more of them to complete it.

 

3. Art is often most impactful at the top of the funnel when you are aiming to make an emotional connection with users. As users advance through the funnel and move out of the discovery mindest (i.e. evaluating what the site has to offer) and become more task-oriented (i.e. add to cart and checkout), aim for a simpler layout that keeps users focused on the primary conversion event.

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5
nolanmargo 07-03-14
 

Re: Ask the Expert your eCommerce questions!

Cheers Khattaab!!

Khattaab 07-03-14
 

Re: Ask the Expert your eCommerce questions!

@JohnH, when collecting user information, minimize the friction that you introduce to the user experience, especially for new visitors. Consider targeting pop-ups only to return visitors because intuition for new users will be to close a pop-up without even reading it.

 

Alternatives to pop-ups that introduce less friction include featuring the text input fields on the page itself or introducing a floating fixed element that reveals from the side rail or bottom of the page the scrolls with the user. These methods of are far less intrusive than a pop-up.

 

When deciding where in the funnel to feature a pop-up or alternative method, avoid making the email sign-up call to action the first element that a user interacts with. Enable users to interact with some site content (and potentially navigating beyond the homepage) so their purpose for navigating to the site is at least somewhat fulfilled before requesting personal information. A trend in e-commerce is requesitng the email address at multiple stages of the funnel, including at the end when a user has advanced to checkout. Email credentials are then requested to create an account in lieu of completing a purchase as a guest.

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5
eranfiliba 07-03-14
 

Re: Ask the Expert your eCommerce questions!

Thanks for taking the time answering questions. 

 

What is the optimum timeframe for an optimizely test? Tests sometimes declare winners, then take them backSmiley Happy Results vary quite a lot in time, and the variations tend to converge at some point in time. 

 

How does Optimizely decide on the 'winner'? The sample size calculator doesn't usually go hand in hand with the optimizely 'winner' deciding system.

 

Thanks!

kzap 07-08-14
 

Re: Ask the Expert your eCommerce questions!

How do you isolate experiments across the site? Like a product page experiment from a checkout experiment. The easiest way is to to use multi page or just 1 at a time? Cause im concerned about experiments affecting other experiments

Level 1
Khattaab 07-08-14
 

Re: Ask the Expert your eCommerce questions!

@eranfiliba,

 

There is no universal truth around how long to run a test; it's all contingent on your traffic volume. This is where the sample size caluculator comes in. Letting a test run indefinitely to wait for a winner does not adequately consider opportunity costs. Defining a minimum detectable effect (MDE) is an essential parameter to establish a level of lift that you are willing to wait for. Best practice is to use the sample size calculator before you run the experiment so that your testing program is standardized and you don't have a sliding scale for experiments that you are tempted to wait-out because there is a near winner.

 

The sample size calculator is an external tool that considers MDE and statistical power; the Chance to Beat Baseline calculation featured on the Optimizely results dashboard is considering only 1-tailed statistical significance. This is why these tools must be used together.

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5
Khattaab 07-08-14
 

Re: Ask the Expert your eCommerce questions!

@kzap,

 

The best way to isolate experiments from one another is to make them mutually exclusive. The procedure for setting-up mutually exclusive experiments is explained in this Knowledge Base article.

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5