Rich-text Reply

statistical significance

ChrisHedick 03-10-17

statistical significance



I have a question around statistical significance. One test, same number of visitors, 2 different goals. The goal with FEWER total conversions and a SMALLER lift has HIGHER statistical significance. How is that possible?


I have also plugged the goal's numbers into an Excel calculator for statistical significance. Optimizely says the goal has 1% significance while the Excel formula says 100% significance.

Screenshots and further explanation are in this Google Doc. 


tomfuertes 03-10-17

Re: statistical significance

My guess is that one is your Primary Metric and one isn't.

Optimizely's Stats Engine will "slow down" non-primary metrics to avoid false positives and help you make better business decisions based on actual anomalies and non-random chance.

You can read more about stats engine here:
- Tom Fuertes | CTO @ CROmetrics / LinkedIn
"Most Impactful Use of Personalization" and "Experience of the Year" Optie award winner.

Experimentation Hero
ChrisHedick 03-10-17

Re: statistical significance

Thanks Tom, unfortunately that is not the case and both are additional metrics and not primary.