Winning Tests: How I Learned to Stop Worrying (About Data) and Love the Bomb
Everybody knows the challenges of creating an effective landing page - it's the page that gets the first hit on your website for a targeted campaign. It's a chance for you to prove your credibility and make a value proposition. It's the page that can make or break your conversion funnel. Figuratively speaking, it's your first date with the customer, and you better not be second guessing whether you chose the chocolates rather than the roses.
Most of us have been there - quibbles about what content gets the spotlight, constant worry over what gets pushed below the fold, and the speculation that follows (even if it has a proven track record elsewhere). We all know relevance and content is key - it's practically Marketing 101. But for any given page, it's easy to forget that you just can't win when you're trying to hit targets with a shotgun/scattershot approach, even if the content is golden. In the process of trying to please everybody, you simply end up appealing to nobody. You can brainstorm countless meetings, create small iterative tests all the way into the new fiscal year, and go over every report in your arsenal to determine your most valuable content; but sometimes, you need to recognize when you should simply hit the 'detonate' button.
I wanted to prove that the key to a winning landing page is about simplicity of design; even if it meant completely dismantling and taking apart the previous design. By removing and de-emphasizing the amount of content that appeared on the page through design, we change the focus of the landing page to simply be about conversion, rather than the building of credibility to lead to a conversion. The changes were drastic - this included the de-emphasis of the hero video asset and putting it below the fold, hiding the form elements off the page and putting them into a modal popup, simplifying the top and bottom navigation, new text copy, and a different visual theme.
Baseline - http://goo.gl/XnJl6O
Variation #1 - http://goo.gl/FYkdZo
Variation #1 (form open) - http://goo.gl/dfbhHL
The test result was a resounding success. With drastic changes, I expected drastic results, and was thoroughly pleased to discover a 45.1% uplift in engagement, and a 633.1% uplift in conversion, with both metrics reaching 98% and higher confidence ratings in a sustained and large sample size.
Sometimes it's easy to become mesmerized by data points, trends, and whether or not your new messaging will affect your relevancy score in Google Adwords, but if we assume that your email or PPC promotions are optimal (which is a completely different beast), it's really about respecting why someone clicked-through in the first place. If someone cared enough to actionably make that first click to end up on your landing page, you really don't need to convince that user any further with additional content or distractions. Just give the customer what they want in the simplest, most attractive user experience you can build, and let them be on their way.
I'm sure some people would say "of course your new variation worked, look how much more attractive it looks", or that my reasoning and approach are sitting on the edge of extremist reductionism. Well, I guess if I was being honest about it - yeah, those people would be 100% correct. A/B testing gives us the ability to be data-driven in our approach, but too often we forget that it also empowers us to take greater risks and liberties in our designs. I'm not saying that we should completely push data aside in favor of instinctual design - I'm just trying to say that a better balance can exist between when you need to leverage your analytics, and when you need to take action directly to your design.
Sure, maybe that text copy we culled was lovingly-crafted for hours to achieve maximum relevance in less than 25 characters. Alright fine, we previously proved a right-column form layout is the optimal way to have a form on the page. Okay, maybe that video content took a great deal of time and resources to create, and has an exceptionally high level of lead participation.
But who's the one holding up the test results with a 633.1% uplift in conversions right now?
Russell Lee is an Online Web Specialist at Maplesoft. He likes wearing a lot of metaphorical-work hats when it comes to internet marketing and design, but doesn't really enjoy wearing actual, real hats.
Thanks! The success is a testament to the overall point I was trying to make - sometimes you need to put the analytics down, and just get out there with something completely new, especially when something's not working well to begin with.
I also understand that we have the fortune of being a lot more agile when it comes to this sort of approach. Not everyone can move as quickly and drastically with developmental changes or if people have to sign-off for messaging and things like that.
Just want to say that my years of a/b testing have taught me that pretty doesn't always win. As you've shown, there is a lot more to it than that. The designers on the Obama campaign must have collectively spent over 200 hours on variations with a better design against an ugly control. We had very limited success just testing prettier pages.
Would you mind posting the raw data for the results? I would love for everyone in the community to start doing this.
Thanks @kylerush !
Haha, we've also had our share of tests where we thought a more attractive page would win, but it either ended up making no visible change, or worse. In this case, it's true, there's far more to it than just making a page 'more attractive'. We could sit all day and dissect all the parts that made our new variation work better, but when it comes down to it, we established a new baseline to optimize from. In fact, that specific page has already had a few more optimization test changes to it since.
As for the raw data, we'd like to protect that information and not divulge our traffic numbers, so I can't do that unfortunately.
"...we forget that it also empowers us to take greater risks and liberties in our designs..."
Couldn't agree more - its easy to get complacent, but having the fearless mentality to get drastic can lead to incredible discovery. Great fearless work!
Hey @HeatherW, while I'm happy that you featured my story in one of your blog posts (http://blog.optimizely.com/2014/05/06/these-experiments-are-fearless/) I'd appreciate it if you made the correction for my name - unless there's a doppelganger of myself out there named Robert, haha.