Rich-text Reply

Difference Interval

zhanglongpv 07-16-15

Difference Interval

Hi,

 

As a newbie to Optimizely I was exploring on how Optimizely calculates winning variation for an experiment.

 

I had a look at this page - https://help.optimizely.com/hc/en-us/articles/200039895-Stats-Engine-How-Optimizely-calculates-resul...

 

Under Difference Interval, it shows two examples - one is Wining Interval and the other one is losing interval.

 

For winning interval, from the article "In the example above, you can say that there is a 97% chance that the bottom variation (in green) will outperform the original, and the difference in conversion rate will be between .29 and 4.33%"

 

I'm wondering how do you calculate .29 and 4.33% as difference interval??

 

Also in the same article it is mentioned that "In this experiment, the observed difference between the original (14.81%) and variation (17.12%) was 2.31%, which is within the difference interval."

 

I would like to understand how did you arrive those numbers - "-2.41% and -1.03%." as difference interval?

 

Since reports are quite important to make business decisions, being a marketer, I would like to understand the each and everything in detail. Would be great if you could explain these to understand it better..

 

Zhang

 

 

Re: Difference Interval

[ Edited ]

Hey Zhang,

I think the best thing is to really dive into some free statistics resources online. However, I can explain some things to give you context of what you are looking at.

Your difference interval is the window in which the difference between the two variations is expected to fall in based on your statistical significance level. In the example you were looking at, the statistical significance was 97%. So that means your P value is .03, and there is a 3% chance that the difference will not be within the interval.

For a certain significance level, there is an associated metric called a Z score. This tells you how many standard deviations away from the mean your distribution is and any number between a positive and negative Z score from your mean is your difference interval.

For 97% significance, your Z score is 2.17. So the high end of the interval is 2.17 standard deviations away from the mean difference between your variations, and vice versa for the low end.

I think that is about as deep as I can go to give you a basic understanding. But feel free to shoot out any specific questions.

-----

Daniel


Digital Marketing Manager - mindbodygreen
Optiverse User Group Leader - NYC
AdamA 07-17-15
 

Re: Difference Interval

Thanks @danielpeskin for your answer! I'll clarify a couple of things, which we've actually updated our Knowledge Base article to reflect. This was incorrect in the existing article, and I want to thank our in-house statisticians for helping to revise.

 

The difference interval is based on your project's statistical significance setting, which is 90% by default. So what we're saying there is that you have a 97% chance that the variation will outperform the original (statistical significance), and a 90% chance that the difference between the original and the variation will be within the difference interval range. So in that example, there's a 90% chance that the variation will beat the original by between .29 and 4.33%.

 

 

As your statistical significance increases, you should generally also see the difference interval "window" narrow, because Optimizely has more data around the likely range of values.

 

 

 

Optimizely

Re: Difference Interval

Oh right my mistake! Thanks for clarifying Adam!
-----

Daniel


Digital Marketing Manager - mindbodygreen
Optiverse User Group Leader - NYC