MDE in Optimizely Sample Size Calculator
Because of the MDE on Optmizely’s calculator I always end up needing a ridiculously large sample size in order to reach Statistical Significance. I mean, I would want to see a change as low as a couple of percent in conversion rate which leads me to need around 80k sample size per variation.
I know it’s important to not determine a winner prematurely. Then again, I could spend months testing one page.
Am I thinking about this wrong or can someone offer insight? I'm relatively new to Optimizely.
It is difficult to offer advice without knowing your traffic levels and test details but I would suggest that you always go to a statistically significant level when calling a winner of your test.
Maybe the tests you are running are not large enough to generate big enough results?
@KerriA using the sample size calulator is an exercise in opportunity cost. Chasing after 2-3% lift numbers can certainly be a viable improvement for your business, but how long are willing to wait to take action? It's not recommended to take action before you achieve the sample size with adequate (i.e. 80%) statisitcal power because you are risking the presence of false negatives skewing the data.
I suggest starting with an MDE of 5-10% to start with a threshold that will identify more dramatic changes of behavior among your users. Any test result, even if statistically significant and powered, should be deemed as inconclusive and encourage you to move-on in your testing program (i.e. iterate on the same experiment with new goals and targeting conditions or try a new test with a more dramatic change).
Director, Experience Optimization | BVAccel
Thanks @Khattaab. After attending the Optimizely Experience I learned more about statistical noise so I understand that smaller improvements may not be improvements at all.
I guess the bottom line is to focus on tests that are more likely to have a larger impact. Easier said than done!