Rich-text Reply

When Do you Launch Experiments? (Day, Time of Day, etc)

Noah 04-16-15

When Do you Launch Experiments? (Day, Time of Day, etc)

Just curious when folks launch their experiments, if they choose to narrow down an exact time-frame. For some reason I find myself inclined to launch Sunday evenings so that I start the week out on Monday morning with clean data. Since I find that our conversions usually follow a pretty common weekly curve, I like to trigger the experiments on the same day each time.

 

It probably matters very little in the long run since you can adjust the reporting dates in the experiment results - just curious if anyone else out there thinks about this kind of stuff!

Noah Shunfenthal
Marketing Automation & Optimization
Level 2

ryanlillis 04-24-15
 

Re: When Do you Launch Experiments? (Day, Time of Day, etc)

Hi @Noah,

 

Interesting question! Generally speaking you're right that it doesn't matter because you can adjust your results view to get at whatever data you like. Additionally, it's very rare that experiements run for such a short period of time that the exact day or hour you launch will have an impact. 

 

That said, the value of consistently launching experiments at the same time/day of week (which is something you can schedule in Optimizely) is that you may start to notice consistent trends about user behavior during those periods. For example, you may notice an initial negative reaction to variations from return visitors during the first few days as they get used to a new design, and by launching the test at a consistent time and day of week you have one less factor to consider when you first start to take a peak at your results.

 

The one rule I've always had about launching tests is not to do it on a Friday. By launching earlier in the week you give yourself a chance to confirm everything is working as expected. 

Optimizely
td_evans 04-25-15
 

Re: When Do you Launch Experiments? (Day, Time of Day, etc)

[ Edited ]

I'm much the same as above, I refuse to launch any complex test on a Friday just because I can't be there to monitor it over the weekend. Also may avoid our biggest trade days for the initial launch for the same reason.

 

Otherwise as long as we run the test for at least 2 full business cycles, we tend to get consistent enough data. As you say you can filter it down after the experiment to see specific day by day data. 

 

A hard rule of launching only on X day could mean a test being ready but not launching, so that's 6 days data you could be missing out on. 

 

Some sites will run an A/B test for just an hour, due to the amount of traffic they get. Wikipedia are one such example, I can see in cases like this why a consistent test time may be appropriate.

Thomas Evans
Technical Product Manager - Conversion @ Secret Escapes

Level 2