Rich-text Reply

Ways to use Optimizely outside of just A/B testing

td_evans 02-16-15

Ways to use Optimizely outside of just A/B testing

[ Edited ]

I'd love to learn how you all are you all using Optimizely creatively?


It’s now been just under a year since we switched from Google Content Experiments to the Optimizely platform for AB testing. In that period we’ve launched almost 300 different experiments, across numerous territories and platforms. The majority of those have been A/B, or MVT, tests with a clear focus on conversion rate optimisation. However we have also utilised Optimizely beyond that, making it a staple of our tech workflow. Below are a few examples:



Drip-fed roll outs

Over the past year we’ve launched some exciting features, which integrate heavily with 3rd party API’s. Using Optimizely, we redirected a percentage of users to the pages featuring the 3rd party content. This allowed us to monitor impact on user behaviour, as well as allowing us to quickly turn off the integration in the event of any bugs.

We started the experiment to 10% of users, eventually ramping up to 100%. Once comfortable that the integration worked correctly, we removed the boolean conditions around the content and served it to all users without the use of Optimizely.


Landing page creation

We initially begun AB testing dynamic keyword insertion, and later customising landing pages to match the keyword bidding on through AdWords. By granting access to all in the PPC marketing team, we used Optimizely to customise the landing page based upon query string. An example is below, where we customised the landing page for Spa keywords. This Optimizely experiment was then run at 100% of traffic, to a custom audience targeted on query string.


The edited landing page



Original landing page





Secret Escapes runs on a number of servers, which means that even a simple hotfix can take a number of minutes to be deployed. On occasion urgent bugs have been spotted, and in the period of deploying a permanent fix, we have used Optimizely to hide/edit the content.

Once example was with the ad serving platform, which suffered a failure – which resulted in empty blocks of white content on the Secret Escapes site. We reacted quickly to hide these through Optimizely, and used CSS to hide the boxes for 100% traffic until the platform had stabilised. The downtime was only 45 minutes, and would likely have been fixed by the time a hotfix had been deployed, so by using Optimizely we were able to quickly react to ensure there was no reduction to user experience.


Minimum Viable Products

Like most startups in 2015, we generally follow the MVP concept. We look to test and validate our hypotheses, as quickly as possible. Generally speaking, any code which needs to go into the application requires a vast amount testing, in-depth code reviews and if it loses, then all of this needs un-doing. By using Jquery and using Optimizely to serve this, we are able to build, control and track our MVPs significantly quicker. We have used free tools such as Google Forms in conjunction with this to build a complete, polished MVP.

In one example we wanted to validate the idea of whether uses would want to be contacted via phone to complete their booking. We used Jquery to serve a dummy button, which opened a lightbox containing a styled Google Form (worth readingthis site for some of the ways you can ‘hack’ Google sheets) which when submitted populated a spreadsheet and sent off confirmation emails to users, hotels and our in-house team. By doing this solely on the front end, we were able to quickly validate the popularity of such a feature.

Once validated, we were able to take our learnings and build a more robust experience for our users inside the SE web application.


An example of an MVP built solely through Optimizely

An example of an MVP built solely through Optimizely


Whilst Optimizely is definitely strongest at AB testing, it’s worth considering these things and giving them a go – if they help speed up your Tech workflow.


You can read the full blog post here

Thomas Evans
Technical Product Manager - Conversion @ Secret Escapes

Level 2

Pauline 02-17-15

Re: Ways to use Optimizely outside of just A/B testing

At Hotwire we have been using Optimizely for 3 things:


- A/B tests with Conversion Rate as a key success metric (traditional optimization experiments with between 2 and 3 variations)

- MVP : We test new features, we use optimizely to "hack" and evaluate demand and engagement from customers

- Promotions: We use Optimizely to run promotions on our website: Badges, special tags, banners or new pictures go live 100% for the duration of the marketing campaign. This is a very easy and flexible way to run these campaigns on our website.


We definitely want to use Optimizely for Landing page creation in 2015, good to hear that it's been working for you!



Product Manager
Level 2
adzeds 02-19-15

Re: Ways to use Optimizely outside of just A/B testing

Great post Tom.

Good to see Secret Escapes making the most of a really valuable tool.

I have used Optimizely in a number of ways over the past year, it mainly focuses around:
- AB testing
- MVT testing
- Bug Fixes
- Running Promotions

However, one other way I have used it is to drive conversions based on real-time data. This is something that I am looking to automate in the long-term but have had some success in doing this through Optimizely.

An example would be on Black Friday/Cyber Monday. I had one screen with real-time analytics data flickering away and another with Optimizely open. I would watch visitor trends through the day and then implement small conversion hacks to help visitors find the relevant content and drive conversions. This meant I could re-order listing pages on the fly to show the offers most likely to convert at a given time based on the conversion data available in real-time. Whilst this was a tough process to manage it did produce results that mean I am looking into making this an automated process.
David Shaw
Level 11
sandeep 03-23-15

Re: Ways to use Optimizely outside of just A/B testing

Great post Tom.

Like this Post.

Sandeep Fofandi
Software Test Engineer.
Level 1
mhopps 05-19-15

Re: Ways to use Optimizely outside of just A/B testing

The cross-browser check is also a very helpful feature to check for bugs in landing pages! (it looks like Optimizely partners with for this functionality.)

Michael Hopps
Optimization Enthusiast

Michael Hopps
Level 2