Rich-text Reply

How Many Concurrent Tests?

bahamagician 01-12-16

How Many Concurrent Tests?



I'm looking for help in planning some experiments.


I want to run several experiments on my item detail page but I'm wondering what's the best way to go about doing it.  


For example, I want to experiment with the "Add to Cart" button colors and I also want to experiement with the "Free Shipping" placement on the page.


Currently, I've been running the "Add to Cart" button color test on a few selected URL's while concurrently running the "Free Shipping" placement experiment (also on the item detail) on another batch of selected URLS.


I'm wondering if it would be better to just run the "Add to Cart" button color test on every item detail page across the board and then after that experiment is complete, then run the "Free Shipping" placement test on every item detail page.


Any thoughts on this?

robertchan 01-13-16

Re: How Many Concurrent Tests?

I would run the Add to Cart and Free Shipping on the same URLs under a multivariate experiment. You can call one tab, Cart Changes, and the other other tab, Free Shipping Changes. Under each tab, make the modifications, and be sure to include original versions of your pages. This way, you'll be able to gather data on all forms of these changes against each other.
Robert Chan

Experimentation Hero
scleveland 01-14-16

Re: How Many Concurrent Tests?

I agree with @robertchan. I think your best option would be to do a multivariate test on all your item detail pages. This way will give you the best combination of "Add to Cart" button color and "Free Shipping" placement.

The downside with a multivariate is it takes longer because there will be more variations, so you have to have patience. But having it on all of your IDP's will move it along faster and give you better results than doing the tests separately.

If you do it separately, your winning color for the "Add to Cart" button and the winning placement of "Free Shipping" may not be the best combination when both are implemented.
JDahlinANF 01-14-16

Re: How Many Concurrent Tests?

Meh... two tests affecting different aspects of the page... I'd run them both at the same time as separate experiments so that you can set one to 100% as soon as it becomes significant.


If you run it as a multi-variate, the best you can do is stop any new users from entering into the loser variations, but users already bucketed will stay in the losing experience.  By running them as separate experiments, you give yourself the option of stopping the experiment and making the winner live for all users (either via a code change or by creating a new experiment with only one variation that implements the winning code).





Khattaab 01-14-16

Re: How Many Concurrent Tests?

Hi, chiming-in from the Optimizely Customer Success team. The first 2 posts were correct to highlight the importance of considering interaction effects to identify the best combination of changes. Whenever you have multiple changes on the same page designed to influence the same action (in this case adding an item to cart), a multivariate test is your best bet. An MVT will require additional traffic, but you can also turn off any combinations you don't want to test at all or that you find are tanking in conversion rate after a reasonable sample size.
Khattaab Khan
Director, Experience Optimization | BVAccel
Level 5