I have started my second A/B test and I'm not seeing the results I would have expected. There is only a 1.5% increase in my new page design vs. my original even though based on heat map data and other data the new version should be way better then the original. Do these results seem aureate?
There is such a drastic difference between the two designs, it just does not make since to me that they would still perform so close to each other.
Thanks for any thoughts or help anyone can provide.
Version 2 (Removed content and everything above the fold)
How long has your test been running?
My guess is you haven't had enough visitors yet to conclude anything. You can check that by looking at the statistical significance status. It must be 90 or 95% (depending on your settings). Anything lower and your test is not done yet.
Thanks you for your input. I'm at 90% statistical significance right now. It's been running for two weeks and has been stuck at around 1.8% increase in activity on my new page. As you can see the two are very different and the new version has all the buttons above the fold. I figured I'd see a lot more activity with the new one. I guess it did not make that big of an impact.
When you're on your optimizely home, click on the settings tab.
There is a big title "Snippet implementation".
Under the script box, there is a smaller title: Statistical Significance Setting.
It usually is set to 90, though 95 is the reference an analytics person would take.
Oh, about the significance level:
I have a running experiment that has a 90% significance level BUT that is the average.
If I take all my goals one by one, one of them has a 99% statistical significance while the two others only have 30%.