Statistical significance does not move
I am running a test that has a statistical significance of 64% since day two or three of the test. It spiked to >99% the first day and dropped to 64% after that.
Now it is not moving for more than two weeks.
How should I interpret this?
Solved! Go to Solution.
If so, then it means that your test has a low "power"... perhaps if it runs for a really really long time you would gain significance, but at his point it appears to be a toss-up. Directional indicators show an improvement, but not "for sure". There are a variety of threads in the Optiverse Community about how to read these scenarios and next steps.
thx for the answer.
Can you shoot me some keywords to search for in the optiverse?
I didn't find anything about my "problem", hence the question :-)
Most of the threads would be located under Strategy & Culture.
A few recent examples - there are definitely a bunch of older ones too that talk about how to handle non-impactful experiments, low traffic issues, and similar: