Statistical Significance is always 0% for all my tests
I don't know why, but statistical significance for all my goals and tests is equal to 0%.
Maybe it is a problem of optimizely, as recently it had some tech issues...
My goals have more than 100 visitors and 25 conversions for each variation, but for some reason it requires more than 100 000 to be sure that it is statistical significant!
Please help... it is not my first problems with optimizely stats, and for now I cannot swear them at all.
Thanks for your help!
Please try to investigate the source of the problem, as it is definitely not issue from our side.
All my experiments were created on Feb 3, after that there were optimizely tech issues with stats, and now I have problem with statistical significance.
Does that mean that all data gathered over the period of the issue is not true?
Please answer and tell us what to do.
So got following materials from Optimizely folks. Apparently any tests after 1/21/2015 will be using this new stats engine...
But unfortunately, I still thinks that there are some problem in their algorithm.
I have simple experiment with 20 000 visitors running 10 days and scientific significance equals 0% for all my goals.
I think, 20 k visitors is COMPLETELY enough for choosing winner at least for Engagement goal!
Otherwise, it requires too many time, traffic and money for testing simple things ( for example - color of the buy button)...
NzarKornitsky. Same here. I have 23K visitors tested and engagement goals are over 3K conversions on each experiments. While one has a pretty consistent lift (conv rate trend line) over control, significance is still 0%... the message still sais <100 visitors required, but I feel like I've been seeing that 'forever'.
There is definitely something wrong!!!
4k visitors over 10 days for this variation - needs additional 98 k visitos = 245 days !!!!
WOW! Testing with new statistics is not super-effective...
Darwish from Optimizely here. Do you mind sending me private messages with the experiment ids of a few of these experiments? We will take a look ASAP.
Product Manager at Optimizely
Yes, sure I can give you experiments ids, so you can check.
It is simple test of the page with pausable/non-pausable video on it.
It has 28 k visitors and still all statistical significance equal 0% and it wants too big number of visitors remaining... so such simple test will take too many time, traffic and money, as my traffic is paid.
At the moment all my tests is paused, as I'm out of the 50 000 limit per month.
I took Optimizely several month ago for testing, and if I get rid of all problems, I will definitely use it, but for now I cannot rely on its results.
Here are some other ids:
Same test but for other page. (I have 4 such tests, all of them have 0% for all goals, I sent you two ids with the biggest number of visitors)
3) Here is other tests - Two pages test
experiment_id=2417580508 - 400 visitors (it is order page test, so not big number of traffic visits it, as it is deep in the sales funnel)
That numbers I got in 10 days of testing, but "visitors remaining" still shows "8600" for sales goal, which is the most important for order page - it is too big number and will take eternity to test it...
Thanks for your reply, I hope you will be able to help me!
Here is a post with a lot of relevant information about this topic!
Sorry for the slow response.
Product Manager at Optimizely