Question on Traditional Statistical Significance vs Stats Engine
If I run a simple A/B test, with one variation and one goal, is traditional statistical significance a more accurate, or innaccurate measure of certainty than Stats Engine?
The new stats engine improves the way to retrieve significant results of your running experiments by reducing the chance of declaring a false positive or negative. Traditional approaches require you to make a hypothesis about your baseline conversion rate and the improvement you are expecting to see in advance. If the baseline conversion rate or the expected improvement turn out to be incorrect you have to recalculate whether your results are actually significant.
Stats engine works sequentially and gives you an evaluation in real time considering the underlying indicators of your current experiment. This means the results you see when using stats engine are valid at every moment in time.
If you are interested in learning more about stats engine I reccomend reading the following articles:
I hope that is helpful. Feel more than free to reply back with any additional questions at all.
EMEA | Amsterdam