Rich-text Reply

A simple effort at personalisation

td_evans 04-21-15

A simple effort at personalisation

[ Edited ]

Personalisation is one of the things I really expect to blow up in 2015. With so much data lying around about our customers, and their behaviour, we now have a great opportunity to optimise their experience by providing more relevant sales and information. Personalisation can be as simple as adding a name to an email subject line, or it can be as complex as completely changing a user experience of a website.

 

In 2015 I’ve been keen to experiment with personalisation of the web, and how we at Secret Escapes present our offers to our users. Currently our ‘homepage’ for signed in users is a list of sales, often 200 long. Whilst we curate and hand-pick every offer that features on the site, we don’t do a great job at creating a personal list for our users. Using Optimizely, and a hack-day, I was keen to implement something that would prove the concept and be sufficient for a very rudimentary A/B test.

 

Our current list of hand-picked sales

Our current list of hand-picked sales

 

Implementation

 

There were multiple ways I looked at using data about our users, as well as how we may be able to quickly create some. I explored 3 different types of personalisation:

 

  1. Search terms. Can we personalise the site and increase the prominence of sales which match previous searches made by the user?

  2. Sale type, based upon most visited sales. We group and tag sales by location and sale type (All-inclusive, Spa). If a user shows a certain propensity per type, for example 80% of sales visited are London based, should we promote those sales?

  3. PPC keyword registration. When we acquire our users via PPC, they are often acquired via a specific keyword. For example “Spa weekend breaks”. If they are looking for these sales, should we promote them for the next x days?

I used Optimizely and Jquery cookie to set a cookie with the above information for the user. We set the Optimizely experiment to run for 100% of users, on the appropriate URLs. For search terms, I stored the last 3 search terms made by a user. For sale type, we incremented a cookie for each sale type match they made. And for PPC keyword we simply set a cookie which stored the keyword that the user reached the site on (this was extracted from a query string).

 

The JS used to set the cookie, this was set in a separate experiment on sign-up page

The JS used to set the cookie, this was set in a separate experiment on sign-up page

 

I then set up multiple Optimizely experiments, and used their targeting to enter users into the appropriate experiment.  If the user matched the cookie, they were entered into an A/B test. The test promoted all relevant sales to the top of the listing page. For example, a visitor who signed up under the keyword of New York, would see the following page:

 

If you signed-up to the site with a keyword featuring NY, you would now see this

If you signed-up to the site with a keyword featuring NY, you would now see this

 

The complexity of this hack would lie in how we treat users who fall into multiple experiments. So for V1 of the experiment we opted to only run with a single cookie type, which was PPC keyword registration. This was the simplest of the implementations, as it wasn’t a dynamic or changing cookie. Once they had signed up it would remain the same until they expired.

 

Results

 

Our initial results are incredibly promising, for the New York keyword we saw the following increases after running for 140 days:

 

Clicks to sale from listing page – +24.4% 
Clicks to purchase page – +30.4%
Completed purchases – +210%

 

Other keywords, including “All-Inclusive” saw very similar results. Seeing significant results for clicks to sale from the listing page.

 

Whilst the numbers on completed purchases are relatively low, all of the KPI’s are positive and it’s something we’re soon to be extending even further across other keywords.

 

Potential problems

 

Naturally this type of personalisation isn’t scalable, however it’s enough to validate an initial hypothesis and spend further time investigating more robust solutions.

 

It’s worth considering the amount of users who will fit into each bucket. The more granular you get with your personalisation, the harder it is to validate the result as significant. We initially chose our highest performing (in terms of signed-up users) keywords, to test good levels of traffic.

 

Cookie compliance. The EU has specific directives on what cookies can be set without opting in, this needs to be taken account and if running this experiment in the EU your users may need to be aware if you are using cookies to personalise their experience

 

http://tech.secretescapes.com/2015/04/ab-testing/a-simple-effort-at-personalisation/

 

 

Thomas Evans
Technical Product Manager - Conversion @ Secret Escapes

Level 2

jonopens 04-21-15
 

Re: A simple effort at personalisation

@td_evans Exciting results as a directional indicator! By search terms, do you mean CPC keyword, site search terms or both sources of data? I think that needs clarifying in the above. Also, are you or have you tried sorting results pages for repeat users based on previously purchased destinations?
Level 1
td_evans 04-21-15
 

Re: A simple effort at personalisation

[ Edited ]

@jonopens We considered doing both, idea 1 was site search keyword and 3 was cpc keyword.

The winning test outlined above was CPC keyword, which the user had registered on.

Optimisation based upon previously purchased has been considered too, along with personalisation based upon local weather. We haven't tested either of these yet. We've only scratched the surface!

Thomas Evans
Technical Product Manager - Conversion @ Secret Escapes

Level 2
mitrakmt 04-22-15
 

Re: A simple effort at personalisation

Great pot! I completely agree with you that personalization will be much bigger in 2015, and already see companies making the move to real-time personalization through data collection. One company, Gigya, has seen huge success in this realm as they help these companies to utilize first party data to then personalize the website/online experience. I actually think this could be really useful for you to look into. Rather than hang curation, a partnerships with Gigya and a front-end personalization platform could solve your problem. From there, it would be extremely easy for you to A/B test the variations (based on the data used) to see which performs better and draws better engagement/conversions.
Level 1
Shanann 04-22-15
 

Re: A simple effort at personalisation

@td_evans this is a great story around personalization! I encourage you to apply for the 'Most Impactful Use of Personalization' Optie Award! https://www.optimizely.com/opticon/opties/
Shanann
Optimizely