Why does the statistical significance stay this low?
Hello friends of the Optiverse I'm running an experiment challenging the steps order of our funnel. Such drastic change has a huge impact! However, even though the uplift is big since day 1, the statistical significance doesn't go beyond 1%.
I use some online tools to have an estimation of the duration and to calculate the sample size or the statistical significance; all of these tools tell me I should have reached the statistical significance by now. So why isn't the case in Optimizely app? Should I rely on Optimizely or the other online tools?
See below the numbers for our main KPI.
Would you consider the variation B as the winner? Would you end the experiment right now?
Any insights on how to deal with this curious situation is welcome!
That is a great question. In short, there is not yet enough volume to know which has won. In the image, there is only 54 conversions (21 and 33). I believe Optimizely actually will not consider something to be stat. sig. until it has at least 25 conversions. I'd recommend waiting a few days. Having more volume will help you avoid false positives.
Hope that helps! I'd be happy to explain in greater detail, if you'd like.