Rich-text Reply

Optimizely Results

PhilLanzarotta 09-23-15

Optimizely Results

[ Edited ]

Hello,

I have started my second A/B test and I'm not seeing the results I would have expected. There is only a 1.5% increase in my new page design vs. my original even though based on heat map data and other data the new version should be way better then the original. Do these results seem aureate?

 

There is such a drastic difference between the two designs, it just does not make since to me that they would still perform so close to each other.

 

Thanks for any thoughts or help anyone can provide.

 

Phil Lanzarotta

 

 

 

 

 

Original Version

 

Original_Version1.jpg

 

Version 2 (Removed content and everything above the fold)

 

Version2.jpg

maria 09-24-15
 

Re: Optimizely Results

Hi Phil,

How long has your test been running?
My guess is you haven't had enough visitors yet to conclude anything. You can check that by looking at the statistical significance status. It must be 90 or 95% (depending on your settings). Anything lower and your test is not done yet.
Level 2

Re: Optimizely Results

[ Edited ]

Hi Maria,

Thanks you for your input. I'm at 90% statistical significance right now. It's been running for two weeks and has been stuck at around 1.8% increase in activity on my new page. As you can see the two are very different and the new version has all the buttons above the fold. I figured I'd see a lot more activity with the new one. I guess it did not make that big of an impact.

-Phil

Re: Optimizely Results

Maria, how do I increase my statistical significance setting?
Thanks
-Phil
maria 09-24-15
 

Re: Optimizely Results

Hi Phil,

When you're on your optimizely home, click on the settings tab.

There is a big title "Snippet implementation".
Under the script box, there is a smaller title: Statistical Significance Setting. 
It usually is set to 90, though 95 is the reference an analytics person would take.

Level 2
maria 09-24-15
 

Re: Optimizely Results

[ Edited ]

Oh, about the significance level:
I have a running experiment that has a 90% significance level BUT that is the average.
If I take all my goals one by one, one of them has a 99% statistical significance while the two others only have 30%.


https://help.optimizely.com/hc/en-us/articles/200039895

Level 2
brianlang 09-24-15
 

Re: Optimizely Results

It would help if you could post additional details about the your KPI goal and supporting metrics.

Brian Lang
twitter.com/cometrico