How Many Concurrent Tests?
I'm looking for help in planning some experiments.
I want to run several experiments on my item detail page but I'm wondering what's the best way to go about doing it.
For example, I want to experiment with the "Add to Cart" button colors and I also want to experiement with the "Free Shipping" placement on the page.
Currently, I've been running the "Add to Cart" button color test on a few selected URL's while concurrently running the "Free Shipping" placement experiment (also on the item detail) on another batch of selected URLS.
I'm wondering if it would be better to just run the "Add to Cart" button color test on every item detail page across the board and then after that experiment is complete, then run the "Free Shipping" placement test on every item detail page.
Any thoughts on this?
The downside with a multivariate is it takes longer because there will be more variations, so you have to have patience. But having it on all of your IDP's will move it along faster and give you better results than doing the tests separately.
If you do it separately, your winning color for the "Add to Cart" button and the winning placement of "Free Shipping" may not be the best combination when both are implemented.
Meh... two tests affecting different aspects of the page... I'd run them both at the same time as separate experiments so that you can set one to 100% as soon as it becomes significant.
If you run it as a multi-variate, the best you can do is stop any new users from entering into the loser variations, but users already bucketed will stay in the losing experience. By running them as separate experiments, you give yourself the option of stopping the experiment and making the winner live for all users (either via a code change or by creating a new experiment with only one variation that implements the winning code).
Director, Experience Optimization | BVAccel