Rich-text Reply

Statistical significance decreasing

bruab 07-01-16

Statistical significance decreasing

Hi there,

 

This may be a silly question, but based on my reading of the Stats Engine blog post, I expect statistical significance to increase with time. I've got some running experiments where it's going up and down, though, so I must have missed something. The post here:

 

https://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age-the-story-behind-optimizelys-...

 

... says "With Stats Engine, statistical significance increases over time" in no uncertain terms. Can someone help clarify this for me?

sequential-testing-graphic2.png
Level 1

CouchPsycho 07-01-16
 

Re: Statistical significance decreasing

To be honest, i did not read the post completely but the sentence itself seems to be irritating. Why?

Because statistical significance depends on user behaviour and comparing two groups ( e.g. original and variation). Since you can not predict behavoiur, mathematics will lead to changing results in both directions.

Maybe the article talks about the difference interval which is explained, when hovering on the graphics.

I did a "few" tests and statistical signifance changed in both directions.



"the essence of the creative act is to see the familiar as strange." (anonymous)

Re: Statistical significance decreasing

[ Edited ]

so shouldn't the knowledge base article reflect that? It clearly states that stat significance increases with each visitor.

CouchPsycho 07-04-16
 

Re: Statistical significance decreasing

Maybe yes. But one of the community managers is in charge now :-)




"the essence of the creative act is to see the familiar as strange." (anonymous)