Rich-text Reply

Experiment showing incorrectly after checks

daveyperks 06-08-16

Experiment showing incorrectly after checks

[ Edited ]

I created an experiment a few weeks ago where I moved an item from one of the tabs in the main menu to another tab within the main menu to see if this would increase CTR to that page. 


Before turning the experiment on I looked at it in preview mode, and used the force parameter and checked in Chrome, IE and Firefox, and it all showed fine so I set the experiment live.


Within one day the experiment has reached statistical significance, after running for 7 days it had increased the CTR by 640%. However I also received two emails from colleagues saying that the experiment was showing strangely to them. (See screen shots).


Screen shot one shows someone just landing on the homepage that is how it appeared, with the drop down menu already down.


Screenshot two shows a blank field at the end of the drop down menu when there should be another link there.


Not only is this frustrating as everything appeared fine to me, I think that it could have skewed the results because I do not know how many people saw the first screen shot where the drop down menu with the variant already opened. If this was the case for a lot of people it could mean the entire experiment is skewed. 


Firstly can someone help me figure out why when I tested it looked fine, but to others it showed differently. Secondly I have decided to duplicate the experiment and run it again (since it wasn't a long running one), but is there any javascript or something that I could use to ensure this drop down issue doesn't occur again and taint my results?

Screen Shot 2016-06-02 at 10.22.44.png
Screen Shot 2016-06-02 at 10.20.29.png
beckyhall 06-09-16

Re: Experiment showing incorrectly after checks

Without seeing the code, it's virtually impossible to tell you what's breaking. I can tell you that it's important to thoroughly test your experiments both pre-launch (preview & force parameters) and post launch (you can launch to a test cookie audience or only to your IP address if you want to test it "live" before other people can see it) and in all of the browsers/versions your website supports.

Some reasons why your experiment might have looked fine to you but not your friends include "race conditions" and test code that is not cross-browser compatible. Race conditions can happen when sometimes Optimizely will try to modify elements before they exist, but other times everything will work as expected. This is particularly problematic with elements that are built through JavaScript or brought in through AJAX and can be remediated by polling for the elements in question before trying to modify them.
robertchan 06-09-16

Re: Experiment showing incorrectly after checks

Your preview may not have been the latest revision you thought you were putting live, and that revision had errors. To check what revision you're on, open console while you're in your experiment's edit mode and type optimizely.revision in the console as you're saving your experiment. You'll see the revision number. Next, force preview your experiment and do the same thing. Check that both revision numbers are the same.

Finally, the visual error can be due to JavaScript, CSS, or both. I'm assuming the display and/or visibility state of your menu is not correct when called. That, or the CSS in your Experiment CSS is intended for one variant, but is running globally causing errors on your other variants.
Robert Chan