Rich-text Reply

What is the additional risk of accepting a false winner when we delete variations?

marie 06-10-15
Accepted Solution

What is the additional risk of accepting a false winner when we delete variations?

[ Edited ]

Does anyone know the exact percentage of additional risk of calling a false winner/loser that we take when we delete a variation from a treatment that has been already running for a certain amount of time?

 

Consider the following examples:

 

Treatment ALPHA

Variations: 

A

B

C

 

Treatment ALPHA adjusted
Variations:
A
C

(B is deleted)

 

Treatment BETA
Variations:
A
B
C

D

E

F

G

 

Treatment BETA adjusted
Variations:
A
C

D
E
F
G

(B is deleted)

 

Treatment GAMMA 

Variations:
A
B
C
D
E
F
G

 

Treatment GAMMA adjusted
Variations:
A
C
D
(B, E, F, G are deleted)

 

 

Level 2

Hudson 06-10-15
 

Re: What is the additional risk of accepting a false winner when we delete variations?

[ Edited ]

Hi @marie,

 

Good question. 

 

You shouldn't be deleting variations from running experiments UNLESS you're confident that the variation in question is certainly a loser. 

 

If we accept the assumption that we're statistically confident that the deleted variation in question is a loser, then the risk of falsely declaring one of the remaining variations a false winner is exactly what it would be normally, or directionally inverse to that winner's measure of confidence (So if we declare 97% significance for the winning variation, the risk of a false declaration would be much closer to 3% than 20%). 

 

However, if you're canceling a given variation before it had been declared to be a loser, then you do, in fact, run a risk of a false positive,  directionally proportional to how confident the declaration that the variations is a loser would be. For example, if you were to cancel variation that was under-performing, but we only declared 50% significance, then we'd have much more risk of falsely declaring another variation a winner if we were to cancel it. That risk would certainly be less than 50%, but it would be higher than what our engine renders as statistically significant. 

 

We can't help you exactly quantify the hypothetical risk of calling a falsely declaring a given variation a winner in the instance that you cancel a variation mid-experiment. Our Stats Engine uses an industry-leading calculation that balances bayesian and frequentist approaches,dynamically employing corrective factors derived from our own historical data. Because of that, extrapolating an exact figure would be impossible for you - truthfully, it depends!. I recommend you read this technical white paper explaining some of the approaches we've taken if you have deeper questions about our calculations.

 

The most practical piece of advice I could give is to never delete an experiment variation mid-experiment unless you're highly sure that the variation in question is a loser. If you follow that advice, then you shouldn't have to worry about falsely declaring winners. 

 

Happy testing,

Hudson

 

 

Optimizely
marie 07-15-15
 

Re: What is the additional risk of accepting a false winner when we delete variations?

Hi Hudson,

Terrific reply, thanks!
And thank you for sharing the paper. Will work through it and probably come back with some additional questions later on. Smiley Wink

Sincerely,

Marie
Level 2