Rich-text Reply

Statistical significance does not move

Sven 11-17-15
Accepted Solution

Statistical significance does not move

Hi guys,

 

I am running a test that has a statistical significance of 64% since day two or three of the test. It spiked to >99% the first day and dropped to 64% after that.

Now it is not moving for more than two weeks.

 

How should I interpret this?

 

Kind regards,

Sven

Level 2

JDahlinANF 11-17-15
 

Re: Statistical significance does not move

I presume the other metrics (visitors, success events, etc.) are moving?

If so, then it means that your test has a low "power"... perhaps if it runs for a really really long time you would gain significance, but at his point it appears to be a toss-up. Directional indicators show an improvement, but not "for sure". There are a variety of threads in the Optiverse Community about how to read these scenarios and next steps.
Sven 11-19-15
 

Re: Statistical significance does not move

Hi nap0leon,

thx for the answer.
Can you shoot me some keywords to search for in the optiverse?
I didn't find anything about my "problem", hence the question :-)
Level 2
JDahlinANF 11-19-15
 

Re: Statistical significance does not move

Most of the threads would be located under Strategy & Culture.

 

 

A few recent examples - there are definitely a bunch of older ones too that talk about how to handle non-impactful experiments, low traffic issues, and similar:

At what point do you give up on your experiments?

How would i report on these inconclusive results?

What to do when Experiments are inconclusive?

Testing with low traffic, probability of no difference or a loss