Changes to control and variation while test is running
If I make a change to my control page and make the same change to my variation page, that shouldn't have an effect on my results, right? If everything still remains constant, but the initial change made for the experiment - I don't see why it would be an issue.
What do you guys think?
Solved! Go to Solution.
It depends on what the changes are... some things create bad user experience...
Like, suppose your page background is white with red buttons and your experiment makes the buttons green.
If you change both control and variation to a green background... well, I'd bet green butttons on a green page don't perfrom very well.
IMO, use your gut - if the changes are inocuous and not related to what you are testing, go for it.
Technically, however, any change could create a UX issue and people who care about being strict in their testing methodologies will tell you to try to control for as many of these other variables as possible.