Statistical significance decreasing

[#1]

Hi there,

This may be a silly question, but based on my reading of the Stats Engine blog post, I expect statistical significance to increase with time. I've got some running experiments where it's going up and down, though, so I must have missed something. The post here:

https://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age-the-story-behind-optimizelys-...

... says "With Stats Engine, statistical significance increases over time" in no uncertain terms. Can someone help clarify this for me?

[#2]

To be honest, i did not read the post completely but the sentence itself seems to be irritating. Why?

Because statistical significance depends on user behaviour and comparing two groups ( e.g. original and variation). Since you can not predict behavoiur, mathematics will lead to changing results in both directions.

Maybe the article talks about the difference interval which is explained, when hovering on the graphics.

I did a "few" tests and statistical signifance changed in both directions.

[#3]

so shouldn't the knowledge base article reflect that? It clearly states that stat significance increases with each visitor.

[#4]

Maybe yes. But one of the community managers is in charge now :-)