Rich-text Reply

Question on Traditional Statistical Significance vs Stats Engine

danielr 04-17-15

Question on Traditional Statistical Significance vs Stats Engine

If I run a simple A/B test, with one variation and one goal, is traditional statistical significance a more accurate, or innaccurate measure of certainty than Stats Engine? 

Level 1

johnnyb 04-17-15
 

.

[ Edited ]
 
Level 1
Mario 04-17-15
 

Re: Question on Traditional Statistical Significance vs Stats Engine

[ Edited ]

Hi there,

 

The new stats engine improves the way to retrieve significant results of your running experiments by reducing the chance of declaring a false positive or negative. Traditional approaches require you to make a hypothesis about your baseline conversion rate and the improvement you are expecting to see in advance. If the baseline conversion rate or the expected improvement turn out to be incorrect you have to recalculate whether your results are actually significant. 

 

Stats engine works sequentially and gives you an evaluation in real time considering the underlying indicators of your current experiment. This means the results you see when using stats engine are valid at every moment in time.

 

If you are interested in learning more about stats engine I reccomend reading the following articles:

 

I hope that is helpful. Feel more than free to reply back with any additional questions at all.

 

Kind regards,

 

Mario 

Sr. Technical Support Engineer
EMEA | Amsterdam
Optimizely