Surprising result on my first A/B test. Is something wrong?
I've set up my first A/B test, which has now been running for 7 days with 90% siginifcance level.
The test is about the positioning of buttons.
In Variation 1 the buttons are much more visible than in the Original. However, for the original the conversion rate of the button click is 40% and in the variation it's 17.5%.
I'm surprised that the less visible buttons are being clicked so many more times. As this is my first experiement I'm wondering if it's possible something could be set up wrong.
Unfortunately I don't think there's a way to attached screen grabs to this post.
Sometimes the results of A/B testing could be quite surprising.
That being said, it could easily be a tracking error, depending how you setup the click goals.
Are the new buttons (more prominent) created by you separately? Any chance the original buttons have a class which appears on multiple places across the page?
For example, if your button (variation A) has a class "btn" and that's the click goal you've set, but that class appears on other elements on the page, this could easily skew your results. I'm just throwing some ideas out there, but without seeing the setup it would be pretty difficult to tell.