Rich-text Reply

Why does the statistical significance stay this low?

Clem a week ago

Why does the statistical significance stay this low?

Hello friends of the Optiverse Smiley Happy I'm running an experiment challenging the steps order of our funnel. Such drastic change has a huge impact! However, even though the uplift is big since day 1, the statistical significance doesn't go beyond 1%.


I use some online tools to have an estimation of the duration and to calculate the sample size or the statistical significance; all of these tools tell me I should have reached the statistical significance by now. So why isn't the case in Optimizely app? Should I rely on Optimizely or the other online tools? 


See below the numbers for our main KPI.


Mon image


Would you consider the variation B as the winner? Would you end the experiment right now?


Any insights on how to deal with this curious situation is welcome!



Level 2

Re: Why does the statistical significance stay this low?

Hi Clem,
That is a great question. In short, there is not yet enough volume to know which has won. In the image, there is only 54 conversions (21 and 33). I believe Optimizely actually will not consider something to be stat. sig. until it has at least 25 conversions. I'd recommend waiting a few days. Having more volume will help you avoid false positives.

Hope that helps! I'd be happy to explain in greater detail, if you'd like.
Parker Sorensen