Rich-text Reply

Statistical Significance is always 0% for all my tests

NazarKornitsky 02-11-15

Statistical Significance is always 0% for all my tests

Hi everybody,

I don't know why, but statistical significance for all my goals and tests is equal to 0%.
Maybe it is a problem of optimizely, as recently it had some tech issues...
My goals have more than 100 visitors and 25 conversions for each variation, but for some reason it requires more than 100 000 to be sure that it is statistical significant!
Please help... it is not my first problems with optimizely stats, and for now I cannot swear them at all.

Thanks for your help!
Regards,

Nazar

kirizawa 02-11-15
 

Re: Statistical Significance is always 0% for all my tests

I'm having a same issue, too. I wish Optimizely team could respond sooner on this...
Level 2

Re: Statistical Significance is always 0% for all my tests

Optimizely Support team, there are several people with same issue.
Please try to investigate the source of the problem, as it is definitely not issue from our side.
All my experiments were created on Feb 3, after that there were optimizely tech issues with stats, and now I have problem with statistical significance.
Does that mean that all data gathered over the period of the issue is not true?
Please answer and tell us what to do.
Thanks!
kirizawa 02-13-15
 

Re: Statistical Significance is always 0% for all my tests

Re: Statistical Significance is always 0% for all my tests

Thanks Kirizawa for the info!
But unfortunately, I still thinks that there are some problem in their algorithm.
I have simple experiment with 20 000 visitors running 10 days and scientific significance equals 0% for all my goals.
I think, 20 k visitors is COMPLETELY enough for choosing winner at least for Engagement goal!
Otherwise, it requires too many time, traffic and money for testing simple things ( for example - color of the buy button)...
kirizawa 02-13-15
 

Re: Statistical Significance is always 0% for all my tests

[ Edited ]

NzarKornitsky.  Same here. I have 23K visitors tested and engagement goals are over 3K conversions on each experiments. While one has a pretty consistent lift (conv rate trend line) over control, significance is still 0%... the message still sais <100 visitors required, but I feel like I've been seeing that 'forever'.

Level 2

Re: Statistical Significance is always 0% for all my tests

My engagement goals are 4k visitors? conversion rate equals almost the same from day to day, significance is still 0%, but the messages says "97200 visitors remaining" (O_O)
There is definitely something wrong!!!
4k visitors over 10 days for this variation - needs additional 98 k visitos = 245 days !!!!
WOW! Testing with new statistics is not super-effective...

Re: Statistical Significance is always 0% for all my tests

I am having the same problem. I have over 900 visitors to some of my variations but zero significance. This appears broken, so i filed a ticket.
darwishg 02-18-15
 

Re: Statistical Significance is always 0% for all my tests

Hey guys,

 

Darwish from Optimizely here. Do you mind sending me private messages with the experiment ids of a few of these experiments? We will take a look ASAP. 

 

Best,

Darwish 

--
Product Manager at Optimizely
Optimizely

Re: Statistical Significance is always 0% for all my tests

Hey Darwish!
Yes, sure I can give you experiments ids, so you can check.
1) experiment_id=2440770098
It is simple test of the page with pausable/non-pausable video on it.
It has 28 k visitors and still all statistical significance equal 0% and it wants too big number of visitors remaining... so such simple test will take too many time, traffic and money, as my traffic is paid.
At the moment all my tests is paused, as I'm out of the 50 000 limit per month.
I took Optimizely several month ago for testing, and if I get rid of all problems, I will definitely use it, but for now I cannot rely on its results.

Here are some other ids:
2) experiment_id=2416600398
Same test but for other page. (I have 4 such tests, all of them have 0% for all goals, I sent you two ids with the biggest number of visitors)

3) Here is other tests - Two pages test
experiment_id=2417580508 - 400 visitors (it is order page test, so not big number of traffic visits it, as it is deep in the sales funnel)
That numbers I got in 10 days of testing, but "visitors remaining" still shows "8600" for sales goal, which is the most important for order page - it is too big number and will take eternity to test it...
Thanks for your reply, I hope you will be able to help me!
Regards,
Nazar
darwishg 03-04-15
 

Re: Statistical Significance is always 0% for all my tests

Hey guys,

Here is a post with a lot of relevant information about this topic!

Sorry for the slow response.

https://community.optimizely.com/t5/Using-Optimizely/Significance-Sample-Sizes-and-Stopping-Tests/td...
--
Product Manager at Optimizely
Optimizely