Test results could be influenced by other tests
We recently ran an experiment to test whether certain changes on the page will lead to higher free signups conversion. We also add in another goal: 'free to paid conversion' just to see if the page may influence users to get the paid version of our app. The results came back positively, but there was a 220% improvement on the 'free to paid conversion' goal, which is not what I was expecting.
I'm curious if the results of this test could possibly be influence by other test that is running at the same time as well. This is because we had several test on different pages running simultaneously.
Our 'free to paid conversion' goal is triggered when a user lands on the complete payment page so we confirm someone bought something if you're on that page. My theory is that a visitor visited Test A variation page and sign up a free account, then several days later stumbled upon Test B and purchase a paid account. So it is Test B that influence visitors to purchase, but the results is also reflected on Test A.
Solved! Go to Solution.
This is a good question and can cover a number of different issues. Some great answers have been provided by my colleagues on posts such as this, here, including links to other articles that they have included in their answers.
To cut a long story short (and to quote my colleague) "unless two tests are affecting similar items, the randomization of bucketing visitors between experiments should mitigate interaction effects in most cases."
If I've misunderstood at all, or you have some more details in the experiment setup that you'd like to share, feel free, and hopefully someone will be able to help out!