Rich-text Reply

Lift is less than MDE

tabiorad 12-28-15

Lift is less than MDE

Hello,

 

In the sample size calculator, if I calculate the size based on a lift number (example: 10%), but the winner in my test only shows a lift of +2% over baseline, can I safely say this was a failure because it didn't reach the +10% that I entered in the calculator? (no other goals associated with the test and sample size was reached)

 

Thanks

 

Level 2

AllisonR 12-30-15
 

Re: Lift is less than MDE

Hi there,

 

The sample size calculator is intended to give you an estimation of the amount of visitors you will need before your test reaches significance. This does not necessarliy mean that once you reach that amount of traffic, your results will have reached the lift you were looking for. Our blog article introducing Stats Engine goes into more details about why "committing to a detectable effect and sample size in advance is inefficient and not intuitive" (https://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age-the-story-behind-optimizelys-...

 

Luckily, because of the way Stats Engine is now calculating your results, you won't need to worry about using the sample size calculator before running your tests. Instead, you'll want to focus on getting statistically significant results from your tests. If you're not seeing the improvement you were looking for after getting statistically significant results, you might be interested in the following section of our support article on statistical significance: https://help.optimizely.com/hc/en-us/articles/200133789-How-long-to-run-a-test#significance

 

Hope this is helpful! 

 

Best,

Allison

Allison
Sr. Technical Support Engineer
Optimizely | EMEA
Optimizely
tabiorad 12-31-15
 

Re: Lift is less than MDE

Thanks Allison

Level 2