Rich-text Reply Realizes a 33% Increase in Sign Ups

davidciccarelli 07-27-14 Realizes a 33% Increase in Sign Ups

[ Edited ]

Optimizely Case Study





Dear Optiverse,


We've been around the testing block a few times, most famously for a case study on the Conversion Rate Experts blog where we realized a 400% increase in sign-ups.  That story was also covered on the Google Website Optimizer blog.


In searching for a new tool, we settled upon Optimizely because it came highly recommended.


Why Optimization Matters


As I've emphasized with our team, we’re resisting being a company-centric organization.  Rather, we continually strive to being customer-centric.  One way we can stay focused on our customers is to look at our website through their eyes.  Then, when we make changes, it's because customers have voted with their clicks. As an organization, we want to challenge the status quo, eliminate guess work and make data-driven decisions.


Variance Testing


Rather than jumping right into split testing, I learned at the Website Optimization conference about the importance of variance testing.  We needed to determine the natural variation in sign-ups on any given day.  By knowing this number, the “variance,” we can eliminate false positives and know with confidence that the number is statistically significant.  As you can see from the variance test, we needed a test improvement greater than 0.4% for us to be confident that the test was indeed accurate and it should be considered conclusive.



Original Hypothesis


The current version of our home page features ten (10) professional voice actors, five (5) male voices and five (5) female voices.  We originally designed the home page this way because we hypothesized that by featuring voice talent on our home page, we would show visitors that we have “real talent" and “real people,” and thus increase conversions.


New Hypothesis


With our intense focus on using the main page of the website to inform visitors and lead them towards signing up for either a free or paid account, we must be willing to challenge the status quo.  The hypothesis was by removing the “Featured Voice Talent” on our home page, we will focus on visitors on the website’s value (is this right for them) and encourage more sign-ups.


The Test


The Home Page testing originally ran for 10 days with 11,258 visitors participating.  A follow-up test ran for 5 days with 6,463.  We set-up “goals” in Optimizely to identify a successful outcome as a sign-up for any level of service with




The Results


Both tests resulted conclusively that by hiding the Featured Voice Talent, we are able to acquire more sign-ups.  One test achieved a 28% increase and the follow-up test yielded a 33% increase, as displayed in the screenshot below.




For the voice talent on the website who may have previously gained some visibility by being featured, they now benefit by having more clients such as video producers, creative directors and advertising executives sign-up, search for and ultimately hire more voice actors - not just the few people previously on the home page.  Now, the Featured Voice Talent are showcased as a series of Top 100 lists.


Key Takeaways


Throughout my decade of experience, I'm continuously reminded that you should test all your changes, big and small. These results in particular taught me that:


  • We should create a hypothesis for each test

  • We should test everything

  • We should be confident in our changes, because they were scientifically proven to increase conversions


Thanks Optimizely!


Kathryn 07-28-14

Re: Realizes a 33% Increase in Sign Ups

Congratulations on the great results David! Thank you as well for sharing your successes with the community and providing these tips; I hope others will benefit from the knowledge you've gained through testing.

Re: Realizes a 33% Increase in Sign Ups

Great post, David. Thanks for sharing it here!
Keith Lovgren