Rich-text Reply

Multiple variants staggered over time

tomneville 11-02-15

Multiple variants staggered over time

(1) I want to confirm that the following is viable:  I want to run a few variants of one page but, since all variants will not be ready immediately (awaiting dev work) and not all will reach significance in the same amount of time, is it possible to: launch one variant now, another in 10 days, and a third in 20 days, changing the traffic allocation assigned to each over time (e.g. if variant 1 reached significance after 15 days, change that to 0%, giving variant 2 50% until variant 3 is setup, at which point I would give them both 33% until one reaches significance). Is that viable?

 

(2) If after variant 1 reaches signficance, I change all future variants not yet started to be variants of variant 1 (rather than of the original), I believe I can compare the future variants to variant 1 (rathe rthan the original) by changing the date range and the baseline in the drop down of the results page to be that variant. Is this correct? 

 

Thanks

Tom

 

JDahlinANF 11-02-15
 

Re: Multiple variants staggered over time

You would want to run each change as a different experiment.

 

Suppose you start with 50/50 between A and B.

After a while - say, 10 days - B wins, so you send 100% to B.

During this time, users previously bucketed into A will continue to see A.

 

After a whlie - say, 10 days - C is ready, so you split traffic 50/50 between B and C.

Only users who are not part of the experiment are split 50-50 between B and C.During this time, users previously bucketed into A will continue to see A.

Users previously bucketed into B will continue to see B.

 

and so on.

 

If you wanted to maintain a true 50-50 split and, for some reason, not change experiment IDs, you would need to leave it at 50/50 until the new varitaion is ready and place the updated code into the losing variation.

 

e.g.,

Suppose you start with 50/50 between A and B.

After a while - say, 10 days - B wins, so you leave everything along until variation C is ready.

When C is ready, edit the A variation so that it runs the new code.

etc.