Statistical significance decreasing
This may be a silly question, but based on my reading of the Stats Engine blog post, I expect statistical significance to increase with time. I've got some running experiments where it's going up and down, though, so I must have missed something. The post here:
... says "With Stats Engine, statistical significance increases over time" in no uncertain terms. Can someone help clarify this for me?
Because statistical significance depends on user behaviour and comparing two groups ( e.g. original and variation). Since you can not predict behavoiur, mathematics will lead to changing results in both directions.
Maybe the article talks about the difference interval which is explained, when hovering on the graphics.
I did a "few" tests and statistical signifance changed in both directions.
so shouldn't the knowledge base article reflect that? It clearly states that stat significance increases with each visitor.