Experiment showing incorrectly after checks
I created an experiment a few weeks ago where I moved an item from one of the tabs in the main menu to another tab within the main menu to see if this would increase CTR to that page.
Before turning the experiment on I looked at it in preview mode, and used the force parameter and checked in Chrome, IE and Firefox, and it all showed fine so I set the experiment live.
Within one day the experiment has reached statistical significance, after running for 7 days it had increased the CTR by 640%. However I also received two emails from colleagues saying that the experiment was showing strangely to them. (See screen shots).
Screen shot one shows someone just landing on the homepage that is how it appeared, with the drop down menu already down.
Screenshot two shows a blank field at the end of the drop down menu when there should be another link there.
Not only is this frustrating as everything appeared fine to me, I think that it could have skewed the results because I do not know how many people saw the first screen shot where the drop down menu with the variant already opened. If this was the case for a lot of people it could mean the entire experiment is skewed.