Rich-text Reply

Incorrect Statistical Significance

danielr 03-12-15

Incorrect Statistical Significance

[ Edited ]

Hi all,


I've been running a test since the 3rd of March and the results are as follows.



Visits: 7330 


Conversion rate: 0.91%



Visits: 7050

Conversions: 87

Conversion rate: 1.23%


The variation has  increased conversion rate by +35%. The statistical signifiance of the test has been hovering at around the 85-88% mark for around 5 days - which is unusual, as the the CR% numbers have remained steady.


However, when I run the control vs variation numbers through external statistical significance calculators (Hubspot, VWO, Get Data Driven), it reaches a significance of 97% of all of them - please see the below.




Can anyone help to explain why this is happening? This is frustrating, as I don't know which numbers to believe and these changes are worth millions to my company - but I need a SS of +90% to bring this to a board meeting, ans the fact that it's hovering in the high 80s in driving me crazy!


Thanks in advance for any insight you can offer, it is much appreciated.




Level 1

Amanda 03-13-15

Re: Incorrect Statistical Significance

Hi Tom. This is a great question and there is a fantastic response to this from Optimizely's Statistician, Leo, in the discussion that you can find here:

This should answer your question, but please let us know if you would like any additional details. Thanks for posting this.