08-14-14

MDE in Optimizely Sample Size Calculator

Because of the MDE on Optmizely’s calculator I always end up needing a ridiculously large sample size in order to reach Statistical Significance.  I mean, I would want to see a change as low as a couple of percent in conversion rate which leads me to need around 80k sample size per variation.

I know it’s important to not determine a winner prematurely. Then again, I could spend months testing one page.

Thanks.

Level 2

Highlighted

Re: MDE in Optimizely Sample Size Calculator

I will go to the statistical significance line in 95% of my tests. Some times I can clearly see a losing test so will jump in and call it early.

It is difficult to offer advice without knowing your traffic levels and test details but I would suggest that you always go to a statistically significant level when calling a winner of your test.

Maybe the tests you are running are not large enough to generate big enough results?
David Shaw
Level 11
Khattaab 08-14-14

Re: MDE in Optimizely Sample Size Calculator

@KerriA using the sample size calulator is an exercise in opportunity cost. Chasing after 2-3% lift numbers can certainly be a viable improvement for your business, but how long are willing to wait to take action? It's not recommended to take action before you achieve the sample size with adequate (i.e. 80%) statisitcal power because you are risking the presence of false negatives skewing the data.

I suggest starting with an MDE of 5-10% to start with a threshold that will identify more dramatic changes of behavior among your users. Any test result, even if statistically significant and powered, should be deemed as inconclusive and encourage you to move-on in your testing program (i.e. iterate on the same experiment with new goals and targeting conditions or try a new test with a more dramatic change).

Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5
KerriA 09-11-14

Re: MDE in Optimizely Sample Size Calculator

Thanks @Khattaab.  After attending the Optimizely Experience I learned more about statistical noise so I understand that smaller improvements may not be improvements at all.

I guess the bottom line is to focus on tests that are more likely to have a larger impact.  Easier said than done!

Level 2