Incorrect Statistical Significance
I've been running a test since the 3rd of March and the results are as follows.
Conversion rate: 0.91%
Conversion rate: 1.23%
The variation has increased conversion rate by +35%. The statistical signifiance of the test has been hovering at around the 85-88% mark for around 5 days - which is unusual, as the the CR% numbers have remained steady.
However, when I run the control vs variation numbers through external statistical significance calculators (Hubspot, VWO, Get Data Driven), it reaches a significance of 97% of all of them - please see the below.
Can anyone help to explain why this is happening? This is frustrating, as I don't know which numbers to believe and these changes are worth millions to my company - but I need a SS of +90% to bring this to a board meeting, ans the fact that it's hovering in the high 80s in driving me crazy!
Thanks in advance for any insight you can offer, it is much appreciated.
This should answer your question, but please let us know if you would like any additional details. Thanks for posting this.