Rich-text Reply

What should I avoid changing while experiment is running?

WilliamHendric 09-30-15

What should I avoid changing while experiment is running?

[ Edited ]

Hi, 

 

Consider this scenario. 

 

I am running ab testing for two pages... ie old page and new design page

 

The experiment has been set up and its running. In between if i make any changes on the new design page, will it affect the results in any way? 

 

Also one more question..when the experiment is set up and running , what are the things should be avoided?

JDahlinANF 09-30-15
 

Re: query on A/B Testing.

1- "if i make any changes on the new design page, will it affect the results in any way. "

Changes certainly could impact the performance of the new design.  It depends on how significant the change is.  Minor tweaks... meh, probably not a big deal (fixing a minor alignment issue, realizing your RGB value of your font was off a smidge and adjusting it to be brand-correct).  But if the change is big - swapping content from one location to another, significantly changing the HERO image to something 'more appealing', etc. - it certainly could impact the performance of the new page. 

 

Consider these two questions "is the change between 'new page' and 'newer page' something that could be A/B tested?"  If the answer is yes, then you should reset the experiment and start over.  If the answer is no, ask whether the change is needed or not ("What is the motivation behind the change?"). 

 

2- Ideally, you don't make any changes to either page during the duration of the experiment.  If the two pages are mostly similar and you are only moving one item on the page, then any changes should be made to both page.  If they are, like, two entirely different designs for a landing page, I would want to avoid tweaking the design on one of them if that same adjustment does not apply to the new page.

 

This is a very broad topic - you may find similar questions and topics in the "Strategy" and "Testing Ideas" sections of Optiverse.

Ehsan 10-06-15
 

Re: What should I avoid changing while experiment is running?

[ Edited ]

Hi @WilliamHendric,

 

In addition to what @nap0leon stated, making changes to the new design page will only directly affect your results if you change an element that is tied to an Optimizely goal.  In other words, if a goal is tracking a specific element, and that element changes or disappears in your update, that goal will stop tracking correctly for the redirect variation.

 

See this article for more info on goal targeting:

https://help.optimizely.com/hc/en-us/articles/200040675-Redirect-experiments-Test-separate-URLs-#cli...

 

Once the experiment is running, the only thing you should avoid changing is the traffic allocation percentages between variations.  The overall traffic (green sliding bar) can be altered without issue, but changing the percent ratios between variations will greatly skew your results.

 

See this article for more info:

https://help.optimizely.com/hc/en-us/articles/200040115#variation_percentage

 

Hope that helps!

Optimizely