Rich-text Reply

Changing total traffic allocation in the middle of a risk-averse experiment, skews results?

Caroleg 06-30-16

Changing total traffic allocation in the middle of a risk-averse experiment, skews results?

I changed traffic allocation on control and my variations mid experiment, now I'm seeing the data go in a complete opposite direction.. Anyone have any experience with that?

Level 1

CouchPsycho 06-30-16
 

Re: Changing total traffic allocation in the middle of a risk-averse experiment, skews results?

Only twice. In both cases i changed the allocation value for at least on variation to zero.

Users stay in their bucket once they have been assigned to it. After setting the allocation value to zero, already bucketed users occasionally hit the goal when they returned to the site. Because of the bucket size keeping the same and conversions could only rise the conversion rate was continiously increasing.

It took me two experiments to realise my "error" :-)



"the essence of the creative act is to see the familiar as strange." (anonymous)