Rich-text Reply

Removing goals during an experiment

oliver-w 03-07-16

Removing goals during an experiment

Hi All,

 

I have read about adding goals during a live experiment, and why it should be avoided: https://help.optimizely.com/hc/en-us/articles/200039895#fdr_control 

 

What about removing goals after an experiment has started? Are there any data integrity issues to bear in mind when doing so? 

 

Any info would be greatly appreciated.

 

Thanks,

Perry

Level 2

Joel_Balmer 03-07-16
 

Re: Removing goals during an experiment

Hi Oliver!

You're right to understand / assume that changing experiments, once started, should be avoided if possible, and understand that there can be issues with doing so.

The issue with adding goals once an experiment has started is that you haven't got data from the beginning, which can introduce bias, misinterpretation, etc. But also due to the way Stats Engine works, your other goals will become more conservative in declaring statistical significance.

Given this, by removing goals your other goals will become less conservative when declaring their significance.

The Optimizely stats page has more links to more details about how this works (false discovery rate control):

https://www.optimizely.com/statistics/

I hope that information helps!
oliver-w 03-07-16
 

Re: Removing goals during an experiment

Hi Joel,

Thanks for your prompt reply.

Trying to interpret what you said about removing goals during a live experiment, are you saying that the Stats Engine will automatically adjust to this change, or that the experiment results should be flushed to preserve data integrity? If it's relevant, the experiment is than a week old so only has a few days' data.

Thanks,
Perry
Level 2