Rich-text Reply

Is it ok to pause a test, and start it again two weeks later?

niki 06-28-16

Is it ok to pause a test, and start it again two weeks later?

We've recently had to roll back minor changes to our site, and as a result have had to pause live tests.

 

We're in a position to launch tests again. 

I wanted to know whether it's ok to unpause these tests, or whether I should duplicate them and launch them as new tests?

 

Our users have a research-purchase lifecycle of between 1 to 30 days. 

 

Thanks

Level 1

CouchPsycho 06-28-16
 

Re: Is it ok to pause a test, and start it again two weeks later?

To be honest, i would not do that. You will lose conversion made within the break, especially if the break is shorter than your lifecycle.

You might copy it but you can also reset the results and start again. This is rather a good idea, if the changes you mentioned have an effect on your users. Your results will be "cleaner" by doing so.

But this is just my opinion :-)



"the essence of the creative act is to see the familiar as strange." (anonymous)
JasonDahlin 06-28-16
 

Re: Is it ok to pause a test, and start it again two weeks later?

Hi @niki,

Based on your customer purchase lifecycle, I recommend against un-pausing your experiment after such a lengthy delay.

 

There may be some wiggle room for experiments where you are using "Visit" conversion metrics instead of "Visitor" conversion metrics.  If you are looking at the Add To Cart rate, you could run your reporting over two time-frames and look only at visits during the periods where the experiment is active. 

If you are looking at "Visitor Conversion", however, you have a decent number of visitors who were placed into an expderiment and then converted when the experiment was off.  Similarly, you have visitors who did their shopping before the experiment began and converted when the experiment was running.  Over a long period of time, those differences "was away" into the background.  But by pausing it and reactivating it, you are doubling the visitors who fall into this condition and also introducing a new condition of "saw experiment, removed from experiment, now back in the experiment".

--Jason Dahlin
Analytics and Testing Guru Smiley Happy
juliofarfan 06-29-16
 

Re: Is it ok to pause a test, and start it again two weeks later?

When we reactive a test after more than a day, we usually duplicate it and start over another one.