Rich-text Reply

Double testing on the same area of my site

daveyperks 04-13-16

Double testing on the same area of my site

I am currently running tests on my checkout funnel on my website using URL targetting. However I have more things within my checking out proceedure that I want to test. So there are two parts of this:

 

1. Is it advisable to run multiple tests on the same area of your site? Is it possible that it could skew results by doing so? For instance doubling up on variants e.g. Test 1 variant 1, as well as Test 2 varient 1? (If that makes any sense).

 

2. Is it even possible?

 

Thanks in advance,

David

robertchan 04-13-16
 

Re: Double testing on the same area of my site

1. It depends on what metrics you're using. Generally, if you're using more than one variant to check whether they have effects on the same type of conversion, it would make sense to set up a multivariate experiment.

2. Yes, you can click on Experiment Type within your experiment editor and select multivariate as your option.
Robert Chan

Mobila 04-20-16
 

Re: Double testing on the same area of my site

Exactly the same thing and I wanted to find out.
I intend to do the same test.
Thanks,

Costin

Level 2
daveyperks 04-21-16
 

Re: Double testing on the same area of my site

I don't understand what you mean in point 1. Can you clarify?
Hudson 04-21-16
 

Re: Double testing on the same area of my site

Hi DaveyPerks,

1. Can you define 'same area of your site'? If you mean the same page, then yes, it is not advisable to run multiple experiments on the same page at the same time. However, if they are on subsequent steps of the funnel, its perfectly OK to run different experiments simultaneously due to the principle of randomization. (Check out this article https://help.optimizely.com/Build_Experiments/Simultaneous_testing_and_mutually_exclusive_experiment...

2. It is possible, but again not necessarily advisable. To clarify, it wouldn't be wrong because of the data (hypothetically should be fine under the randomization principle), but because of the potential for individual users getting weird experiences, and because it will take even longer to reach statistically significant conclusions by splitting your traffic into progressively smaller parts. If you have enough traffic, you could consider using Mutually Exclusive JS code. Usually only very large sites do this, as a way to improve test velocity, because they have enough traffic to support it to the degree that '1 test, 1 page' is actually a constraint to test velocity.
Optimizely
daveyperks 04-22-16
 

Re: Double testing on the same area of my site

Hi Hudson,

Many thanks for clarifying.

As I said in my first post, I am running a test on my checkout funnel (to see if removing our header will increase conversions), however I am not getting the number of visitors I was expecting and it is taking much longer for the current URL targeted experiment to draw firm conclusions. I am using a substring match on my basket page, and my main measurement is the number of visitors that 'complete purchase' by landing on the thank you page (also measured as a substring match).

As I said the numbers are coming in very slowly and I know that we do a lot more visitors than are being recorded by looking at the GA.

I have some pressure to get results and also more (different) tests that I wish to run on the whole checkout funnel, but don't want to run them while my initial one is running incase they affect the results.

Thanks.