Rich-text Reply

test landing page comparing 1 week vs the next week

astromana 03-27-15

test landing page comparing 1 week vs the next week

I have a new design for a landing page I wan to test. The new design is so complex, it can't be done within optimizely and it can't be a different url. I want to run the original landing page for a week and then run the new page for a week. then compare the two periods and see which one did better. How do I do this?

Re: test landing page comparing 1 week vs the next week

Hi there,

Thanks for reaching out to the Optiverse!

Depending on how complex these changes are, you might be able to run this as a standard A/B test. Can you ask your developer to send you the code changes that are made? These code changes can be pasted directly into the Edit Code box for Variation 1.

Here's more information on using the Edit Code box.

Hope that helps some! Looking forward to hearing other's thoughts on how best to perform this experiment as well. Smiley Happy


Amy Herbertson
Customer Success
astromana 03-28-15

Re: test landing page comparing 1 week vs the next week

The new page is too complex. It involves page logic being done with settings in the DB and take place before optimizely is even considered on page. I really need to test the page one week with the old design, then test the page for another week with the new design. Is this possible?
greg 03-31-15

Re: test landing page comparing 1 week vs the next week

May I ask why even use Optimizely for this? If you have some analytics tracking in place, such as Google Analytics, it's very simple to compare two time ranges.
Level 2
Hudson 04-01-15

Re: test landing page comparing 1 week vs the next week

Hi Astromana,

Thanks for sharing your question!

While I'd probably still be curious to see that this experiment couldn't be run via a redirect between your control and your more complex variation (as a new page built by you, to your specifications), you raise an interesting topic about the nature of a/b testing vs. before & after comparisons.

One of the reasons why we love a/b tests is that by testing two variations against each other, in real time, is that we effectively control for all external factors that could skew results toward a given variation. (This is assuming that we have a meaningful large volume of traffic!) Before and after comparisons, by contrast, are prone to being skewed by external factors such as a media campaign, news releases by your competitors, or shocks to the economy like a holiday or fluctuation in oil prices.

You mentioned that you'd like to run each variation for a week and then compare; beyond making each variation subject to any external factor that could skew results over those weeks, you'd also be left with a very difficult decision to make in trying to decide how long to run each test, without the help of Optimizely's Stats Engine!

Hypothetically, you might believe that you'll see such strong differences that sophisticated statistics or controlling for external factors may be uneven dart - if so, feel free to apply a post-hoc comparison to your original, but be sure to caveat your interpretation of those results:
a. Traffic volume and quality may be substantially different between different time periods
b. External events will not be controlled for
c. Statistical comparison will need to be applied manually

These caveats all indicate that a before and after comparison will be naturally less powerful for inferring the magnitude and direction of performance differences, and therefore make your business decision less sound.

So, I'd encourage you to share more about the construction of your variation with the community, and confirm that it's absolutely impossible to test your new variation via a redirect.

Good luck! Happy testing.