Rich-text Reply

statistical significance

ChrisHedick 03-10-17

statistical significance

Hello,

 

I have a question around statistical significance. One test, same number of visitors, 2 different goals. The goal with FEWER total conversions and a SMALLER lift has HIGHER statistical significance. How is that possible?

 

I have also plugged the goal's numbers into an Excel calculator for statistical significance. Optimizely says the goal has 1% significance while the Excel formula says 100% significance.

Screenshots and further explanation are in this Google Doc.

 

 

https://drive.google.com/open?id=0B_CrQ9KxLBRkdFI0cTF3OHpvUDQ 

 

tomfuertes 03-10-17
 

Re: statistical significance

My guess is that one is your Primary Metric and one isn't.

Optimizely's Stats Engine will "slow down" non-primary metrics to avoid false positives and help you make better business decisions based on actual anomalies and non-random chance.

You can read more about stats engine here: https://www.optimizely.com/statistics/
____
- Tom Fuertes | CTO @ CROmetrics / LinkedIn
"Most Impactful Use of Personalization" and "Experience of the Year" Optie award winner.


ChrisHedick 03-10-17
 

Re: statistical significance

Thanks Tom, unfortunately that is not the case and both are additional metrics and not primary.