Is it possible to change a control/original during an A/B test

[#1]

I have an experiment running on my website but I have been asked to slightly change the copy on the control. Would this have a knock on effect on the variation? Its a minor copy change.  If it is ok to make the change should I pause the experiment or leave it running?

[#2]

From a purist perspective, you ought not alter anything once an experiment has started.

Though we may endeavor to do our best to test concepts in isolation, the reality, though, is that websites are dynamic living things with content that changes all the time regardless of whether an experiment is happening or not.  In that regard, there is no harm in making "minor copy changes".

But, without knowing details, only you can truely answer the question as to whether you should expect this to have an effect on the control group's performance.  However, you could run a pre-post analysis of the control group's KPIs (click through rate, etc.) to see if the change had any immediate impact.

[#3]

I would rerun the experiment with the change. instead of modifying it while your experiment's running. By not rerunning the experiment with the change, you risk skewing the data as new users to your experiments would get bucketed into this change, but your other user segment wouldn't.

[#4]

@robertchan - Playing devil's advocate here to get a feel for why a "minor" change would necessitate restarting the experiment and set back a testing cycle by several weeks.

For the sake of discussion... let's flesh this out:

1- Let's presume the "minor copy change" is in a portion of the page that is removed in the variation (otherwise the change would be in both and it's a non-issue).

2- The change is indeed "minor" (changing a testimonial to include "Mr" or "Mrs" before their name, or something else that is similarly inconsequential).

How is this any different than if the udpate were performed by the DEV teamor by a CMS update?

Would you advocate re-starting the experiment just because the site content updated or is the important thing the *concept* that is being tested, rather than the specifics.

For example, our HERO spot is typically a slider.  But, sometimes we test it as a vertically stacked panel.  If the UX team decides to change the arrows from white to beige (a minor change) on the slider, do I need to restart the experiment?  The concept of the experiment is not affected (slider vs stacked).

Also, since these experiments may run for several weeks, but we seldom have the same creative up for more than 3 days in a row, would I need to restart the experiment every time the content changes?

[#5]

@nap0leon - Valid points. For me, it's all really dependent on the amount of traffic a website receives along with the amount of time it takes to build statistically significant traffic. Even with the two points you've mentioned, the "minor" change is still dependent on the scope of the experiment and whether more weight is being emphasized locally or globally. In the case that experiments typically run off 1% to 5% traffic allocation, rerunning the change OP stated doesn't necessarily provide any further changes to results, but as with all testing, "doesn't necessarily" also cannot substantiate the opposite outcome. Finally, depending again on the scope, if the copy is being changed for article headlines, the impact of what would be considered a slight change in other experiments could prove consequential. My stance is that the result of a comparison of an experiment's traffic allocation to typical traffic allocation, and scope determine whether an experiment should be rerun. As for other departments, the scope of their focuses are separate from a tester's and therefore comparing updates is non-correlative.

[#6]

Thanks folks, for some reason I didn't spot your replies until today but they are still very helpful. In the end I did not make the copy change as even though it was a minor copy change it was part of the banner that we were using as the control and it could have impacted on the results. Thanks again for the advice.