A/B testing on completely different frontends/backends
At my workplace, we've been working on creating an entirely new version of our existing site, which is a horribly unwieldy Drupal-based beast that was developed before my time. This new version is in Rails, has a fairly different flow, and structure.
From what I understand, A/B testing is meant for comparing small variants of a UI, and that it's not really appropriate for such an enormous comparison (I hesitate to even refer to the new project as a "variation" of the control). However, Optimizely was the tool that the team had been eyeing to test with--would it be appropriate to use it in our situation? Is this even still considered "A/B testing"? If so, what approach would we have to take with using Optimizely? The closest thing I could find was to use the redirect test feature, but I'm not sure that it's enough to accomplish this.
Solved! Go to Solution.
It is difficult to give a 100% safe answer to this question without knowing exactly how your new site compares to your old one. If the same page hierarchy and call to actions are the same then I would use the redirect test.
Hey @cmarguel ,
I would echo the response @adzeds had so greatly provided. From the intial sounds, a redirect experiment would suffice, but as @adzeds also pointed out - more information would be helpful in coming up with a best direction!
In general, when comparing more complete pages against one another, a redirect experiment would be the way to go. More info on that can be found here: https://help.optimizely.com/hc/en-us/articles/2000
Manager, Technical Support
Thanks for the reply. Upon giving it a try, it does look like redirect tests will work, but is it really necessary to have the same page hierarchy? I do know which pages and actions on one site correspond to which actions on the other site, but I wouldn't say there was a 1:1 correspondence.
My main concern with the redirect test--I'm unfamiliar with how this part of Optimizely works--is whether or not I'd risk the users being redirected between sites in the middle of their purchasing, which is undesirable.
I've got it working such that visitors to the root of our domain may get selected to use the new version of our site. Is there a chance that a customer might get selected to use a different variation of the site if they return to the root (which is a use case for both versions of the site)?
You can consider your redirect test here as simply testing two different designs of a website, as opposed to standard A/B test where the two pages have mostly 1:1 correspondence (except for the minor copy / CTA changes, etc. that you're testing for).
If a user is redirected from www.yoursite.com to your new site, www.yoursite.com/newsite, then they have been bucketed into this variation. If they then return to www.yoursite.com, they will continue to see the variation that they have been bucketed into previously, i.e. www.yoursite.com/newsite. They will see this version persistently, until you either stop the experiment or they delete their cookies.
However, you need to take into consideration where this test will run. If you run it only on the homepage and they are redirected, and from thereon navigating around the new version keeps them in the new version (including while purchasing), then there **bleep** back in the original.