Rich-text Reply

Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Amanda 09-22-14

Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

[ Edited ]

The live portion of Ask the Expert is now closed. Please submit your question on the apporpriate discussion forum so our other community members can help you out. 

 

Respond to this post with questions you may have, bounce ideas around, and get to know Chris Goward! He'll be answering any questions you have all week long. Plus, a lucky participant in this discussion will win a free copy of Chris’s book “You Should Test That.”

 

Our Expert:

 

Chris Goward has developed conversion rate optimization strategies for clients such as Google, Electronic Arts, SAP, Magento and BuildDirect.com. He is the brains behind the LIFT™ and Kaizen™ methods and speaks at conferences and seminars around the world to evangelize how marketers should optimize their marketing to maximize their leads, sales and profit. He wrote the book, "You Should Test That," which redefines CRO and shows how to create dramatic business improvements.

 

Learn more at: http://widerfunnel.com, http://youshouldtestthat.com, @chrisgoward. You can also find Chris in Optiverse under the username @chrisgoward

 

Chris Goward_Headshot.jpg

 

Share this opportunity with your friends!

Tweet: Live Q&A with @ChrisGoward going on now. Join me in the discussion http://ctt.ec/PGa_d+

 

 

You can ask questions until EOD September 26th, at which point this session will be closed.  

 

--

If you are interested in being featured as an Expert for a specific topic, please email optiverse@optimizely.com

Also, if you have any ideas for future Ask the Expert Themes, please post them in the Community Ideas board here.

 

 

Optimizely

Jacob 09-22-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi Chris,

Thanks for taking the time to answer the community members questions - I'm sure many appreciate it!

Some of us has been discussing how to set up the best testing documentation structure (http://community.optimizely.com/t5/Strategy-Culture/What-is-your-testing-documentation-structure/m-p...

What's your take and how do you handle that over at Widerfunnel?

Thanks,

Jacob
Level 2

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Thanks for lending your ear Chris. 2 topics I'd love your opinion on:

1. What to do when you get statistically significant conversion differences in an A/A variance test?

2. What to do when 2 systems of record report different results on test performance? (E.g. I've had Optimizely and Google Analytics report vastly different results and had a really hard time making a judgment and justifying my case to stakeholders.)

Would love your input. Thanks!
Amanda 09-22-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hey @chrisgoward - Really excited for this disucssion. Oftentimes, in hindsight you can figure out how different groups of traffic respond to different content and page designs. As an example, @MJBeisch actually just told a great testing story describing how the presence seasonal traffic drastically affected the experiment results. 

 

How can you anticpate the segments that might work for a certain experiment so you can accurately report on this and interate on your test hypothesis effectively? Do you have a process for establishing segments / do you use any specific tools to find this information? 

 

 

 

Optimizely
Highlighted

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi @chrisgoward ,

 

As you know testing without a large volume of traffic is really difficult because of how long it takes to reach statistical relevance.

 

There are a lot of small businesses who might only have 25K unique visitors a month. These might be family run businesses etc and the only way they can afford to test is to do it themselves.

 

Beyond having dramatic differences in test pages, what advice would you have for them on statistical relevance of 95% which could take a very long time to reach. I think there's a sizeable segment of businesses in this position and it'd be great to hear the opinion of someone who seems to take a very common sense approach to testing.

 

Thanks in advance!

  

Keith Lovgren

chrisgoward 09-22-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Yes, there are some good ideas on that thread already, @Jacob . All of the details discussed there should be documented for reference after the test. 

 

I would add that a GANTT chart can help as part of the testing plan. We use these a lot at WiderFunnel to ensure rapid-fire testing from one opportunity to another, so we don't lose time between tests. While one test is running, another 2-3 are in production, ready to launch.

 

You'd need to know the follow-up hypotheses to be tested, so you can incorporate them into the future experiments.

 

Keep in mind that you should avoid overlapping tests within the same funnel as it will invalidate results when mutual exclusion is not an option. 

--
Learn more: http://www.widerfunnel.com/blog
hernan_cas 09-22-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi @chrisgoward ,

 

Thanks for taking the time to answer the community members questions!

 

Focus

 

 

  • Where do you think the effort should be dedicated? 

 

       a) Iterate on small improvements  (+1%, +2% on conversion rate) 

       b) Research to find strong hypothesis that will make big improvements  (>5%)

 

Process

 

 

  • I think a key point is to run the correct tests. The ones that will make your conversion really grow. So, Which is the best approach/process to define which things should be tested? 

e.g:

  • Do exploratory analysis to get some insights. 
  • Analyse the main barriers and promoters and iterate on tha
  • Define which is the final state you want to reach and start making small tests..

Approach 

 

Small changes vs Bold changes .  

 

Sometimes I find that small changes also represents small impact on our KPIs. However they are really good to learn and understand what's happening. On the other-side bold changes may have a bigger impact, but sometimes it is really hard to understand the why and in case you get bad results you will find yourselve lost.

 

I think both are valid based on the context.

 What do you think about this topic?  

 

Targeting

 

Cool Tips for Targeting?

 

Example:

Common Segments:

 

  • New User vs Returning User
  • Traffic Source (Direct, Branded, SEM, SEO)
  • Engaged user 

 

Measuring the Results

 

What would you recommend when making decisions to define a winner:

 

a) Focus on the outcomes (Conversion Rates)

b) Understand the behaviour metrics (CTR,Bounce Rate, etc) and see how that impacts on your outcome metrics.  Learn from then behaviour and decide the winner based on the outcome

 

Qualitative Metrics / Long Term Impact

Deciding on long term impact projects or qualitative impact projects

 

Sometimes there are things hard to measure through and A/B testing tool like engagement or awareness. How do you deal with this kind of projects? Do you A/B test this kind of projects?

 

Thanks a lot!

 

Hernan Castagnola

chrisgoward 09-22-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

@Merritt , to answer your first question:

 

That’s a sticky situation, which is usually caused by a technical setup issue. While there might be statistically significant differences in conversion rates, look at the spread total conversion count too. Are we talking about 50 vs 75 conversions or 5000 vs 7500? That’s more likely to happen with small counts due to statistical clumping. Remember that a 95% statistical significance is still a *probability* and there’s always a chance for strange anomalies.

 

If there is a truly large spread in conversions, try:
1) Re-running the experiment to validate
2.) Cross-browser test to make sure that there wasn't some unknown abnormality in the second version
3.) Look for particular traffic segments (broken down by source, browser type, etc) to see if a variation is getting a high percentage of one traffic source?

 

In virtually every case where we’ve looked into an issue like this, there’s a technical setup error.

--
Learn more: http://www.widerfunnel.com/blog
chrisgoward 09-22-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Your second question raises a very common issue, @Merritt . I’ve asked one of our Senior Strategists, Alhan Keser, to help weigh in with WiderFunnel's latest steps to work with this. He'll post some tips shortly. 

--
Learn more: http://www.widerfunnel.com/blog
Alhan 09-22-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi @Merritt

To answer your second question:

One thing to start with would be to get your Optimizely data and GA data to match as much as possible. To do so, the following steps will help (but will not solve ALL discrepancies):
1.) Place your optimizely script as close as possible to your GA snippet
2.) Whatever filters/advanced segments you're using on your GA view, try to replicate them in your Optimizely audience
3.) Make sure you're tracking the same goals Smiley Happy
4.) Make sure your Custom Variable slot is not being used by another Optimizely experiment or anything else.
5.) Wait long enough that you've got a LOT of data to go by. It could be that there is a discrepancy at first and that it's simply due to lack of traffic volume.

Hope that helps!
Alhan Keser, Optimization Strategist @ WiderFunnel

Product Manager of Liftmap - Plan, track and share your Optimizely experiments with anyone.

Optimizely Platform Certified
Level 2
JohnH 09-23-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi @chrisgoward - I'm curious to learn how your team brainstorms and works together to analyze pages/conversion blockers and come up with testing ideas. Can you tell me what this process looks like at WiderFunnel? Do you have weekly cross functional tem meetings etc?

Level 2
chrisgoward 09-23-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Good question,  @Amanda!

 

There’s a lot of debate about segmentation. There are two general approaches we take:
1. Start with segments already defined
2. Discover segments post-test

 

In the first case, where you already believe you know your audience segments, we will often recommend testing to validate whether they actually respond differently. This is a structured “top-down” approach to segmentation, where you should design tests with the segments already in mind.

 

Or, you may know you have segments but don’t know how to identify them. With companies in this situation, we’ll define activity triggers on the website that imply the segment, then bucket visitors who take that action and then we can test and/or target segment-relevant messages to them.

 

In the second case, which is what you’re talking about, post-test analytics helps to identify groups that respond differently during a test. We’ve often found surprising segments that way, but it involves exploratory analytics with an open mind, rather than looking for pre-defined segments.

 

The seasonal urgency example is something different entirely. I don’t consider that a segment because the same visitors will respond differently at different times due to external urgency.

 

We’ve seen many examples of this. WiderFunnel actually just posted a seasonal urgency A/B test case study showing how a surprise typhoon affected the test results for the International Rescue Committee’s donation campaign landing pages. We learned that different page designs work better during high and low external urgency periods, which was hugely important for maximizing donations for the IRC. 

--
Learn more: http://www.widerfunnel.com/blog
chrisgoward 09-23-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

I get that question at almost every conference, @keith_lovgren. Thanks for raising it here. 

 

The truth is that testing with a lot of traffic is simply more fun Smiley Happy But, not every company has that luxury. 

 

Some consultants even say that it's not worth testing if you have lower traffic like this. I disagree. Low traffic does not mean low revenue and there’s no downside to testing.

 

You already know you need to test dramatically different variations, which is true. But, the dramatic changes don’t necessarily need to mean major design changes. Instead, think about how to make, what I call, large “cognitive changes.” By that I mean changes that impact the six conversion factors, especially the value proposition. Offers, calls-to-action, positioning, etc. can make big impacts.

 

This approach also works well with small businesses who cannot necessarily afford to pay for design and development of variations that require significant design changes.

 

However, there is no way around the wait time on a/b tests with low traffic. Here are the top tips:

  • Limit the number of variations to 1-3 at most (plus the control)
  • Test dramatic cognitive changes
  • Get advice on your variations from experts who’ve tested similar situations to have a better chance of a win (i.e. don’t start from scratch)
  • Test isolations for learning on your highest traffic pages, then confirm on lower traffic pages with simple 1 variation A/B tests
  • Prune under-performing variations more aggressively than normal
  • Be prepared to stop tests, infer potential learning, and move on to confirm that learning with fewer-variation follow-up tests
  • Consider whether a lower confidence interval is a valid business decision. In some cases, it can make sense to go with 90%, taking the extra risk as a tradeoff against the opportunity cost of running fewer tests.

 

--
Learn more: http://www.widerfunnel.com/blog
ben 09-23-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Howdy Chris,
Question 1
We always try to iterate on our tests to see if we can squeeze more out of them. Is that something you would recommend?
Question 2
We have been testing our site for almost a year now and most of the low hanging fruit has been picked. To look in the GA to find actionable data to find places to test is getting harder. Do you have any tips for useful reports/views to run in GA to find overlooked pinch points?
Question 3
I like to use data to figure out what to test. Soon I am going to be asked to test an brand new rebuild of a site so I will not have any historical data to look at. Could you give me some suggestions on how you would think about testing a new site?

Thanks
Ben
(PS I heard you speak at #CTAconf2014 a couple of weeks ago and enjoyed it very much)
Ben Cole
Optimization Co-ordinator at Widerfunnel

ben
Level 2

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Fabulous answer, Chris. Thanks for posting!!  

Keith Lovgren

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi @chrisgoward - I'm trying to find an effective way of documenting results of tests for the small business I work for.

Do you recommend any particular tool or medium for documenting these tests, in particular that doesn't add complexity to the task of documenting, and that is easy to read/navigate to encourage others to read the results? You tips and advice is most welcome!
Stephen Hamilton
Digital Marketing Manager - QIS Packaging
www.qispackaging.com.au
chrisgoward 09-24-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Wow,  @hernan_cas ! That's an ambitious set of questions. 

 

1. Where to dedicate time: without question, spend more time on research for bigger improvements. Your design/development time is fairly equal whether you research or you don't so, make it worth your team's time. AND, you're more likely to get results faster. So overall, you win. Incremental, small changes should be tested if they’ll produce a useful learning, but spending too much effort on tiny changes is what gives conversion optimization a bad name.

 

2. As for your Small vs Bold question; how to find what changes matter: when you find yourself uncertain about what page elements have an impact, you should test that! Make major changes to or remove individual elements from the page to evaluate user sensitivity to those elements. Once you zero in on the elements that make an impact, you know where to focus. Follow-up with experiments isolating those elements that appear to get people to behave drastically.

 

3. Segmentation: see my response to @Merritt above.

 

4. Always consider the winner by only looking at your final point of conversion. End of story. Bounce rates, click-through rates, time on site can all be misleading. Motivating a visitor to transact is the goal. By all means, look at the other metrics to devleop hypotheses for the next round, but those aren't conversion goals. 

 

5. Testing qualitative questions. It is possible to take a scientific approach to nearly everything. Sometimes, more qualitative questions take creativity in how to define the goal and test segments, but it's possible. We've tested for major Fortune 500 brands to determine how different presentations affect purchase intent in long-term decision-cycles, for example. Using a survey tool like Qualaroo, it is possible to a/b test branding changes and collect feedback (in the form of thumbs up/down, or subtle knowledge questions). It does require more traffic as fewer than 3% of visitors are likely to fill out the survey form. 

 

--
Learn more: http://www.widerfunnel.com/blog
chrisgoward 09-24-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi @JohnH 

 

Our cross-disciplinary team uses the LIFT Model as a heuristic framework to analyze pages and develop hypotheses. We use a 4-person team on each project to identify conversion barriers, convert them into hypotheses, refine the test variations and create the test plans. All experiment plans are also reviewed with the larger strategy team to put all ideas through the wringer and make sure they're going to give us the best chance of getting test wins as well as important insights. 

--
Learn more: http://www.widerfunnel.com/blog
chrisgoward 09-25-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

Hi, @ben. Thanks for the kind words about my presentation! It was a fun event to speak at. 

 

1. Iterating on tests: Absolutely! We run many follow-up rounds on any one experiment. This is partly a product of us having so many ideas in the first round that we can't fit into one experiment that we spread out across multiple experiments, but ideas also flow from either the opportunity at hand or other areas of the website that we're testing that we want to re-validate and expand on in other tests.

 

For example, we gain an insight on your product page, we want to bring that insight to the category page. While we're testing there, another nuance comes to light, which we bring back to the product page, and soon we're validating an insight across the website.

 

In each round, we also often include one variation that's a bit "out there" - one that will give a completely unknown result as the hypothesis has not yet been validated. This way, we make sure to cycle through a variety of hypotheses without getting hung up on what we might at the time think is the big idea, when we just hadn't tried something better yet. There's always a higher global maxima out there that we're striving towards.

 

2. Tips for new ideas:

a. Have your grandmother visit the website and watch what happens.

b. Do a user testing study

c. Hire a consultant for an outside perspective

 

3. The good news: you'll have plenty of ready-to-pick low-hanging fruit to go after with the new website. The bad news: it's a crapshoot as to what insights you can re-apply to this website from the previous version. It will come with its own set of strengths and weaknesses, unless it is very similar in design and content. What made a big impact in previous tests may no longer be of such high influence on user behavior. It may take some time to find what users are more/less sensitive to. Be sure to start with a few experiments that try to answer that very question: what part of this website really matters?

 

It sounds like you're already moving down the redesign path, but we're often finding it's more effective to avoid the radical redesign altogether. Redesigns typically end up over-budget, late, and frustrating... and they usually hurt conversion rates and sales. Instead, we're helping our clients redesign their site dramatically through testing. Through a carefully-planned series of A/B tests, we can often redesign the site faster, with fewer headaches, and get better results, without the risk. I call it "Evolutionary Site Redesign" or ESR. 

 

 

 

 

 

--
Learn more: http://www.widerfunnel.com/blog
chrisgoward 09-25-14
 

Re: Ask the Expert: Chris Goward joins us to discuss the Art and Science of A/B testing

As traditional as it may seem, a tool like Excel can do a pretty good job for a small business, @StephenHamilton.

 

When sharing a results overview, just ensure the following minimum requirements are there:

  • Visuals of what you tested. 
  • Test data from experiment ran - visitors, conversions, improvement, statistical confidence
  • Estimated impact on revenue

You could include multiple rounds of testing on one sheet. Have separate sheets for different opportunities. Have a summary page to show all results combined and another sheet to compile all learnings and reference those to the right sheet.

 

There are a variety of tools that can do all of the above, but we often go with spreadsheets as they allow formulas to be used and customized for unique business models, whereas with any other tool, you'll have to have a calculator open all the time and are more likely to make errors carrying over data from one tool to another.

 

More important than the tool, is how you present the results with your team. Make sure that it requires little concentration to understand what is going on.

--
Learn more: http://www.widerfunnel.com/blog