Rich-text Reply

A/B testing on completely different frontends/backends

cmarguel 01-05-15
Accepted Solution

A/B testing on completely different frontends/backends

[ Edited ]

Hi,

 

At my workplace, we've been working on creating an entirely new version of our existing site, which is a horribly unwieldy Drupal-based beast that was developed before my time. This new version is in Rails, has a fairly different flow, and structure.

 

From what I understand, A/B testing is meant for comparing small variants of a UI, and that it's not really appropriate for such an enormous comparison (I hesitate to even refer to the new project as a "variation" of the control). However, Optimizely was the tool that the team had been eyeing to test with--would it be appropriate to use it in our situation? Is this even still considered "A/B testing"? If so, what approach would we have to take with using Optimizely? The closest thing I could find was to use the redirect test feature, but I'm not sure that it's enough to accomplish this.

 

 

 

Level 2

adzeds 01-05-15
 

Re: A/B testing on completely different frontends/backends

Looking at this I would say a redirect test is likely to be the best option.

It is difficult to give a 100% safe answer to this question without knowing exactly how your new site compares to your old one. If the same page hierarchy and call to actions are the same then I would use the redirect test.
David Shaw
Level 11
Brian_Abad 01-05-15
 

Re: A/B testing on completely different frontends/backends

Hey @cmarguel ,

 

I would echo the response @adzeds had so greatly provided. From the intial sounds, a redirect experiment would suffice, but as @adzeds also pointed out - more information would be helpful in coming up with a best direction!

 

In general, when comparing more complete pages against one another, a redirect experiment would be the way to go. More info on that can be found here: https://help.optimizely.com/hc/en-us/articles/200040675-Redirect-experiments-Test-separate-URLs-

 

---
Brian Abad
Manager, Technical Support
Customer Success
Optimizely
cmarguel 01-06-15
 

Re: A/B testing on completely different frontends/backends

Hi,

Thanks for the reply. Upon giving it a try, it does look like redirect tests will work, but is it really necessary to have the same page hierarchy? I do know which pages and actions on one site correspond to which actions on the other site, but I wouldn't say there was a 1:1 correspondence.

My main concern with the redirect test--I'm unfamiliar with how this part of Optimizely works--is whether or not I'd risk the users being redirected between sites in the middle of their purchasing, which is undesirable.

I've got it working such that visitors to the root of our domain may get selected to use the new version of our site. Is there a chance that a customer might get selected to use a different variation of the site if they return to the root (which is a use case for both versions of the site)?
Level 2
Joel_Balmer 01-06-15
 

Re: A/B testing on completely different frontends/backends

Hi @cmarguel,

You can consider your redirect test here as simply testing two different designs of a website, as opposed to standard A/B test where the two pages have mostly 1:1 correspondence (except for the minor copy / CTA changes, etc. that you're testing for).

If a user is redirected from www.yoursite.com to your new site, www.yoursite.com/newsite, then they have been bucketed into this variation. If they then return to www.yoursite.com, they will continue to see the variation that they have been bucketed into previously, i.e. www.yoursite.com/newsite. They will see this version persistently, until you either stop the experiment or they delete their cookies.

However, you need to take into consideration where this test will run. If you run it only on the homepage and they are redirected, and from thereon navigating around the new version keeps them in the new version (including while purchasing), then there **bleep** back in the original.