Lift is less than MDE
In the sample size calculator, if I calculate the size based on a lift number (example: 10%), but the winner in my test only shows a lift of +2% over baseline, can I safely say this was a failure because it didn't reach the +10% that I entered in the calculator? (no other goals associated with the test and sample size was reached)
The sample size calculator is intended to give you an estimation of the amount of visitors you will need before your test reaches significance. This does not necessarliy mean that once you reach that amount of traffic, your results will have reached the lift you were looking for. Our blog article introducing Stats Engine goes into more details about why "committing to a detectable effect and sample size in advance is inefficient and not intuitive" (https://blog.optimizely.com/2015/01/20/statistics-
Luckily, because of the way Stats Engine is now calculating your results, you won't need to worry about using the sample size calculator before running your tests. Instead, you'll want to focus on getting statistically significant results from your tests. If you're not seeing the improvement you were looking for after getting statistically significant results, you might be interested in the following section of our support article on statistical significance: https://help.optimizely.com/hc/en-us/articles/2001
Hope this is helpful!
Sr. Technical Support Engineer
Optimizely | EMEA