Rich-text Reply

Validating if I am running my experiment correctly?

neveen 11-19-14

Validating if I am running my experiment correctly?

I have set up an experiment where I am testing if users are more prone to notice and click on a tag bubble or tag text?  I have two goals, one for engagement and one for click. My experieent has been running for 3 days, with about 10K unique visitors -- yet it still says its inconclusive. I realize that could potentially be a possibility, but i am also questioning if I've set things up appropriately. Is there a way to validate the correctness of my experiement?

 

 

Level 1

MartijnSch 11-19-14
 

Re: Validating if I am running my experiment correctly?

Hi Neveen, not really. The best way is to make sure that you've tested all the variants, you excluded your own IP address for reporting purposes. If you really want a second pair of eyes you could share some more details here so we can help you with anything!
greg 11-19-14
 

Re: Validating if I am running my experiment correctly?

Most likely the change you are testing is too small to show any significant difference, even after 10k visitors. See my post on the Optimizely blog about neutral/inconclusive results and how to learn from them: http://blog.optimizely.com/2014/10/30/the-problem-with-ab-testing-success-stories/

Level 2
neveen 11-20-14
 

Re: Validating if I am running my experiment correctly?

how do i include my IP address? Is this a setting I enable or is a mere paste of my experiment URL?
Thank you Martijn
Level 1
MartijnSch 11-20-14
 

Re: Validating if I am running my experiment correctly?

Exclude ;-), you can exclude it via the IP filtering settings on the project page: http://take.ms/mGEd0

greg 11-20-14
 

Re: Validating if I am running my experiment correctly?

Quick FYI for those reading: IP filtering excludes your actions from being counted in the sample size and results, but it does not exclude you from seeing the experiment.

Also, the IP exclusion field accepts Regular Expressions (aka, regex). So you can exclude multiple IP's, for instance, as follows:

194\.66\.82\.11|22\.231\.113\.64

The "|" character is an OR operator, as in "this one OR this one." The backslashes are an escaping character to specify that the dots are actually part of the IP, and not a wildcard symbol.
Level 2