06-10-15

# What is the additional risk of accepting a false winner when we delete variations?

[ Edited ]

Does anyone know the exact percentage of additional risk of calling a false winner/loser that we take when we delete a variation from a treatment that has been already running for a certain amount of time?

Consider the following examples:

Treatment ALPHA

Variations:

A

B

C

Variations:
A
C

(B is deleted)

Treatment BETA
Variations:
A
B
C

D

E

F

G

Variations:
A
C

D
E
F
G

(B is deleted)

Treatment GAMMA

Variations:
A
B
C
D
E
F
G

Variations:
A
C
D
(B, E, F, G are deleted)

Level 2

Hudson 06-10-15

## Re: What is the additional risk of accepting a false winner when we delete variations?

[ Edited ]

Hi @marie,

Good question.

You shouldn't be deleting variations from running experiments UNLESS you're confident that the variation in question is certainly a loser.

If we accept the assumption that we're statistically confident that the deleted variation in question is a loser, then the risk of falsely declaring one of the remaining variations a false winner is exactly what it would be normally, or directionally inverse to that winner's measure of confidence (So if we declare 97% significance for the winning variation, the risk of a false declaration would be much closer to 3% than 20%).

However, if you're canceling a given variation before it had been declared to be a loser, then you do, in fact, run a risk of a false positive,  directionally proportional to how confident the declaration that the variations is a loser would be. For example, if you were to cancel variation that was under-performing, but we only declared 50% significance, then we'd have much more risk of falsely declaring another variation a winner if we were to cancel it. That risk would certainly be less than 50%, but it would be higher than what our engine renders as statistically significant.

We can't help you exactly quantify the hypothetical risk of calling a falsely declaring a given variation a winner in the instance that you cancel a variation mid-experiment. Our Stats Engine uses an industry-leading calculation that balances bayesian and frequentist approaches,dynamically employing corrective factors derived from our own historical data. Because of that, extrapolating an exact figure would be impossible for you - truthfully, it depends!. I recommend you read this technical white paper explaining some of the approaches we've taken if you have deeper questions about our calculations.

The most practical piece of advice I could give is to never delete an experiment variation mid-experiment unless you're highly sure that the variation in question is a loser. If you follow that advice, then you shouldn't have to worry about falsely declaring winners.

Happy testing,

Hudson

Optimizely
marie 07-15-15

Hi Hudson,