Is it possible to change a control/original during an A/B test
I have an experiment running on my website but I have been asked to slightly change the copy on the control. Would this have a knock on effect on the variation? Its a minor copy change. If it is ok to make the change should I pause the experiment or leave it running?
From a purist perspective, you ought not alter anything once an experiment has started.
Though we may endeavor to do our best to test concepts in isolation, the reality, though, is that websites are dynamic living things with content that changes all the time regardless of whether an experiment is happening or not. In that regard, there is no harm in making "minor copy changes".
But, without knowing details, only you can truely answer the question as to whether you should expect this to have an effect on the control group's performance. However, you could run a pre-post analysis of the control group's KPIs (click through rate, etc.) to see if the change had any immediate impact.
@robertchan - Playing devil's advocate here to get a feel for why a "minor" change would necessitate restarting the experiment and set back a testing cycle by several weeks.
For the sake of discussion... let's flesh this out:
1- Let's presume the "minor copy change" is in a portion of the page that is removed in the variation (otherwise the change would be in both and it's a non-issue).
2- The change is indeed "minor" (changing a testimonial to include "Mr" or "Mrs" before their name, or something else that is similarly inconsequential).
How is this any different than if the udpate were performed by the DEV teamor by a CMS update?
Would you advocate re-starting the experiment just because the site content updated or is the important thing the *concept* that is being tested, rather than the specifics.
For example, our HERO spot is typically a slider. But, sometimes we test it as a vertically stacked panel. If the UX team decides to change the arrows from white to beige (a minor change) on the slider, do I need to restart the experiment? The concept of the experiment is not affected (slider vs stacked).
Also, since these experiments may run for several weeks, but we seldom have the same creative up for more than 3 days in a row, would I need to restart the experiment every time the content changes?