Rich-text Reply
Highlighted

Why does the statistical significance stay this low?

Clem 03-13-19

Why does the statistical significance stay this low?

Hello friends of the Optiverse Smiley Happy I'm running an experiment challenging the steps order of our funnel. Such drastic change has a huge impact! However, even though the uplift is big since day 1, the statistical significance doesn't go beyond 1%.

 

I use some online tools to have an estimation of the duration and to calculate the sample size or the statistical significance; all of these tools tell me I should have reached the statistical significance by now. So why isn't the case in Optimizely app? Should I rely on Optimizely or the other online tools? 

 

See below the numbers for our main KPI.

 

Mon image

 

Would you consider the variation B as the winner? Would you end the experiment right now?

 

Any insights on how to deal with this curious situation is welcome!

Cheers,

Clem

Level 2

Re: Why does the statistical significance stay this low?

Hi Clem,
That is a great question. In short, there is not yet enough volume to know which has won. In the image, there is only 54 conversions (21 and 33). I believe Optimizely actually will not consider something to be stat. sig. until it has at least 25 conversions. I'd recommend waiting a few days. Having more volume will help you avoid false positives.

Hope that helps! I'd be happy to explain in greater detail, if you'd like.
Best,
Parker
Parker Sorensen