Rich-text Reply

Incorrect Statistical Significance

danielr 03-12-15

Incorrect Statistical Significance

[ Edited ]

Hi all,

 

I've been running a test since the 3rd of March and the results are as follows.

 

Control

Visits: 7330 

Conversions:67

Conversion rate: 0.91%

 

Variation

Visits: 7050

Conversions: 87

Conversion rate: 1.23%

 

The variation has  increased conversion rate by +35%. The statistical signifiance of the test has been hovering at around the 85-88% mark for around 5 days - which is unusual, as the the CR% numbers have remained steady.

 

However, when I run the control vs variation numbers through external statistical significance calculators (Hubspot, VWO, Get Data Driven), it reaches a significance of 97% of all of them - please see the below.

 

SSCalc.png

 

Can anyone help to explain why this is happening? This is frustrating, as I don't know which numbers to believe and these changes are worth millions to my company - but I need a SS of +90% to bring this to a board meeting, ans the fact that it's hovering in the high 80s in driving me crazy!

 

Thanks in advance for any insight you can offer, it is much appreciated.

 

 

 

Level 1

Amanda 03-13-15
 

Re: Incorrect Statistical Significance

Hi Tom. This is a great question and there is a fantastic response to this from Optimizely's Statistician, Leo, in the discussion that you can find here: https://community.optimizely.com/t5/Using-Optimizely/Stats-engine-results-vs-classical-stats-results...

This should answer your question, but please let us know if you would like any additional details. Thanks for posting this.

Optimizely