A/B Testing and Personalization at the same time?
A question came up in my team whether it is possible or recommendable to run A/B tests AND personalization campaigns at the same time or not. Most of our A/B tests have macro sales goals like # of Transactions or Conversion Rate, so theoretically a personalization campaign on a high traffic homepage in a prominent space would make some noise within the A/B testing results wouldn't it? What would you recommend?
I'm a believer in being able to do personalisation and A/B testing at the same time. But what probably a lot of people are messing up is that they don't take into account what the personalisation changes have for effect on the analysis + A/B test. So as long as you take the changes into account for personalisation in the analysis of your tests then it should work out fine. What's happening is that you're A/B tests is turning itself because of personalisation into a MVT so you also have to take care of this in the analysis.
Hope this makes sense!
That makes definitely sense and those are also points we had in mind, BUT since Personalization can be seen as a replacement for playing same content for all customers, the ideal situation would be to personalize all of the content areas. And if you have a couple of campaigns running on the homepage, the category pages and perhaps on some landing pages it becomes quite difficultto take all these personalization tests into account while analyzing an A/B test in the checkout process for instance.
So how would you solve that? One option could be to integrate the personalization into the testing roadmap (the available slots) but this would drive the personalization ad absurdum because normally you want to personalize content all the time?