Rich-text Reply

Count statistical significance in Optimizely

wonderz 02-17-15
Accepted Solution

Count statistical significance in Optimizely

[ Edited ]

Hello,

I think I have an issue with counting of statistical significance in Optimizely. 

All counter apps shows me that I have significant results on 90% and 95% significance level. 

For example http://abtestguide.com/calc/ or excel file from http://bit.ly/17O10jc 

But in Optimizely I see only 73% of statistical signifcance on 90% significance level.

 

Do you know how to Optimizely count this metric?

 

My values are:

A: visits 174 conv. 74
B: visits 169 conv. 90

 

Thank you very much.

Joe

Level 1

Joel_Balmer 02-18-15
 

Re: Count statistical significance in Optimizely

Hi Joe!

 

Optimizely has recently changed the way we test, from a normal 'fixed horizon' test to 'sequential hypothesis' testing. Other A/B testing tools (such as ABtestguide's calculator) will calculate based on the old method, which is why calculating using other tools will give you a different result. Our blog post on this explains some of these points, here.

 

If you use our stats calculator, here, this will give you a better calculation because our calculator is powered by the new 'Stats Engine'. However, this calculator is an estimate also, because it makes an assumption that the 'confidence (difference) interval' will be +/- the same value, because it's not possible to guess the confidence interval before results have come in.
The best estimate to use, that takes this into account, is the estimate you see in the results page. It takes confidence interval and the stats engine calculator into consideration.

 

I hope some of that information helps!

Leo 02-18-15
 

Re: Count statistical significance in Optimizely

[ Edited ]

Hi Joe,

 

Just a couple comments to piggy-back on Joel’s response and flesh out some of the changes with Stats Engine.

 

You can still read and interpret your Optimizely significance results in the same way as you did before. The difference is now they will be accurate regardless of whether or not you used a sample size calculator to fix a sample size, and how many times you looked at your results.

 

In general, our significance numbers will be a little lower than ones using fixed horizon statistics, like the ones on the links in your post. This is because Stats Engine accounts for the added risks of looking at your results more than once and testing multiple goals and variations at once (if you do this).

 

In addition to the blog post that Joel mentioned, here is a link to a product update and FAQ on Optiverse https://community.optimizely.com/t5/Product-What-s-New/Product-Update-FAQs-A-new-Stats-Engine-to-pow.... Within you’ll also find further links to a knowledge base article on how Optimizely calculates statistical significance, and a technical write-up detailing the math behind Stats Engine.

 

Thank you for your question and interest.

Leonid Pekelis
Statistician at Optimizely
Leo
Optimizely
wonderz 02-19-15
 

Re: Count statistical significance in Optimizely

Thank you guys for quick response. I will read recommended articles.

Regards.
Joe
Level 1