Rich-text Reply

Changes to control and variation while test is running

jamonthill 05-03-15
Accepted Solution

Changes to control and variation while test is running

If I make a change to my control page and make the same change to my variation page, that shouldn't have an effect on my results, right? If everything still remains constant, but the initial change made for the experiment - I don't see why it would be an issue.

 

What do you guys think?

 

Thanks

 

Jamon

JDahlinANF 05-04-15
 

Re: Changes to control and variation while test is running

It depends on what the changes are... some things create bad user experience...

 

Like, suppose your page background is white with red buttons and your experiment makes the buttons green.

If you change both control and variation to a green background... well, I'd bet green butttons on a green page don't perfrom very well.

 

IMO, use your gut - if the changes are inocuous and not related to what you are testing, go for it.

Technically, however, any change could create a UX issue and people who care about being strict in their testing methodologies will tell you to try to control for as many of these other variables as possible.

jamonthill 05-12-15
 

Re: Changes to control and variation while test is running

Good points. Thanks for the answer, Napolean.

jamonthill 05-12-15
 

Re: Changes to control and variation while test is running

*nap0leon, sorry.