Rich-text Reply

Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

rlee 04-29-14

Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

[ Edited ]

Everybody knows the challenges of creating an effective landing page - it's the page that gets the first hit on your website for a targeted campaign. It's a chance for you to prove your credibility and make a value proposition. It's the page that can make or break your conversion funnel. Figuratively speaking, it's your first date with the customer, and you better not be second guessing whether you chose the chocolates rather than the roses.

 

Most of us have been there - quibbles about what content gets the spotlight, constant worry over what gets pushed below the fold, and the speculation that follows (even if it has a proven track record elsewhere). We all know relevance and content is key - it's practically Marketing 101. But for any given page, it's easy to forget that you just can't win when you're trying to hit targets with a shotgun/scattershot approach, even if the content is golden. In the process of trying to please everybody, you simply end up appealing to nobody. You can brainstorm countless meetings, create small iterative tests all the way into the new fiscal year, and go over every report in your arsenal to determine your most valuable content; but sometimes, you need to recognize when you should simply hit the 'detonate' button.

The Hypothesis:
I wanted to prove that the key to a winning landing page is about simplicity of design; even if it meant completely dismantling and taking apart the previous design. By removing and de-emphasizing the amount of content that appeared on the page through design, we change the focus of the landing page to simply be about conversion, rather than the building of credibility to lead to a conversion. The changes were drastic - this included the de-emphasis of the hero video asset and putting it below the fold, hiding the form elements off the page and putting them into a modal popup, simplifying the top and bottom navigation, new text copy, and a different visual theme.

 

Baseline - http://goo.gl/XnJl6O

Variation #1 - http://goo.gl/FYkdZo

Variation #1 (form open) - http://goo.gl/dfbhHL

 

The Results:
The test result was a resounding success. With drastic changes, I expected drastic results, and was thoroughly pleased to discover a 45.1% uplift in engagement, and a 633.1% uplift in conversion, with both metrics reaching 98% and higher confidence ratings in a sustained and large sample size.

 

Sometimes it's easy to become mesmerized by data points, trends, and whether or not your new messaging will affect your relevancy score in Google Adwords, but if we assume that your email or PPC promotions are optimal (which is a completely different beast), it's really about respecting why someone clicked-through in the first place. If someone cared enough to actionably make that first click to end up on your landing page, you really don't need to convince that user any further with additional content or distractions. Just give the customer what they want in the simplest, most attractive user experience you can build, and let them be on their way.

 

I'm sure some people would say "of course your new variation worked, look how much more attractive it looks", or that my reasoning and approach are sitting on the edge of extremist reductionism. Well, I guess if I was being honest about it - yeah, those people would be 100% correct. A/B testing gives us the ability to be data-driven in our approach, but too often we forget that it also empowers us to take greater risks and liberties in our designs. I'm not saying that we should completely push data aside in favor of instinctual design - I'm just trying to say that a better balance can exist between when you need to leverage your analytics, and when you need to take action directly to your design.

 

Sure, maybe that text copy we culled was lovingly-crafted for hours to achieve maximum relevance in less than 25 characters. Alright fine, we previously proved a right-column form layout is the optimal way to have a form on the page. Okay, maybe that video content took a great deal of time and resources to create, and has an exceptionally high level of lead participation.

 

But who's the one holding up the test results with a 633.1% uplift in conversions right now?

 

-RL

 

Russell Lee is an Online Web Specialist at Maplesoft. He likes wearing a lot of metaphorical-work hats when it comes to internet marketing and design, but doesn't really enjoy wearing actual, real hats.

Level 2

HeatherW 04-29-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

Really incredible results (633.1%, WOAH!) and beautiful new design.  Nice work.  This is a must see @Cara  & @kylerush 

Optimizely
rlee 04-29-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

[ Edited ]

Thanks! The success is a testament to the overall point I was trying to make - sometimes you need to put the analytics down, and just get out there with something completely new, especially when something's not working well to begin with.

I also understand that we have the fortune of being a lot more agile when it comes to this sort of approach. Not everyone can move as quickly and drastically with developmental changes or if people have to sign-off for messaging and things like that.

Level 2
MeganBush 04-29-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

Awesome results! Big changes can lead to big rewards - Congrats! The new version definitely looks better, and while there are a lot of elements that changed and could affect the results, the overall thought process behind it tells you something very important about your customers: they don't need as much convincing as you might think to convert. Thanks for sharing!
Megan Bush
kylerush 04-30-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

Thanks for posting this, Russell! Great write up.

Just want to say that my years of a/b testing have taught me that pretty doesn't always win. As you've shown, there is a lot more to it than that. The designers on the Obama campaign must have collectively spent over 200 hours on variations with a better design against an ugly control. We had very limited success just testing prettier pages.

Would you mind posting the raw data for the results? I would love for everyone in the community to start doing this.

Thanks again!
Optimizely
zemaniac 04-30-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

Best title ever! ahaha
Alessio Romito
Project Manager at 21DIAMONDS GmbH
Level 2
rlee 04-30-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

[ Edited ]

Thanks @kylerush !

Haha, we've also had our share of tests where we thought a more attractive page would win, but it either ended up making no visible change, or worse. In this case, it's true, there's far more to it than just making a page 'more attractive'. We could sit all day and dissect all the parts that made our new variation work better, but when it comes down to it, we established a new baseline to optimize from. In fact, that specific page has already had a few more optimization test changes to it since.

As for the raw data, we'd like to protect that information and not divulge our traffic numbers, so I can't do that unfortunately.

Level 2
Jonathan 04-30-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

The version without the form looks VERY appealing! Love the space in it. Smart.
Level 2

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

Great share Russell - Thanks!

"...we forget that it also empowers us to take greater risks and liberties in our designs..."

Couldn't agree more - its easy to get complacent, but having the fearless mentality to get drastic can lead to incredible discovery. Great fearless work!
rlee 05-08-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

[ Edited ]

Hey @HeatherW, while I'm happy that you featured my story in one of your blog posts (http://blog.optimizely.com/2014/05/06/these-experiments-are-fearless/) I'd appreciate it if you made the correction for my name - unless there's a doppelganger of myself out there named Robert, haha.

Level 2
HeatherW 05-08-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

@rlee - I apologize profusely! I have a dear friend named Robert Lee and I got it stuck in my head. The article is fixed, thank you for calling it out. I owe you a swag bag Smiley Happy
Optimizely
rlee 05-08-14
 

Re: Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb

Haha no worries! My co-workers and I thought it was pretty funny - they've all now relegated to calling me Robert this week.
Level 2