0 Likes

Launch at 100%

Status: Great Idea!
by cubelodyte ‎05-27-2014 May 27, 2014

Today, my Development Director asked my to turn on the winning variation 100% until the release at the end of the week.

Although I've been generally inclined to handle this manually, through the Experiment Editor, I decided to give the new Results view/funtionality a ride.

 

To my surprise, the result of "launching" the winning variation variation was to pause the other variations in the experiment.

What it didn't do, was kick up the trafffic for the experiment to 100% (as I had expected). Instead, it left the experiment at its overall traffic setting. 

 

This, I suppose, is a great topic for discussion. In most occasions, I find that software which assumes too much in order to be "helpful" is usually confusing and assumes "too much". 

Strangs how, when it comes to something I want, it no longer seems like assuming too much. It seems like the feature fell short. 

 

This, of course, reveals the subjective nature of such decisions and casues me, as a Usability Analyst, to fall back on my normal "Less is More" foundational principles. 

Conclusion: I agree with the decision, but am frustrated with the outcome.

 

Since my experience is limited enough that I don't easily imagine a use for "Launcing" at anything other than 100% traffic, I would love to see some additional comments here to expose the contrasting views.

 

Meanwhile, I can envision a variation of this feature that allows you to choose the lauch percentage from the confirmation dialog.

Status: Great Idea!
Level 6

Comments
by Dodiet
‎05-27-2014 May 27, 2014

When using Optimizely to drive 100% of the traffic to a variation while waiting for the release development to be completed, I have found it best to duplicate the experiement and then allocate 100% of the traffic to the winning variation you found in this new duplicated experiement. This is done in addition to pausing the old experiement.

 

By doing it this way, it puts website visitors into new buckets that will then always show them the winning variation. If you simply reallocate the existing experiement's traffic, it will only affect new visitors that have not seen the experiement yet. Previous visitors will still see the variation they were initially assigned to.

 

This may solve the problem that you found when it didn't kick up the traffic for the experiement to 100% as you expected.

Level 1
by Optimizely
‎05-28-2014 May 28, 2014
Hi cubelodyte!
Dodiet described how to best use Optimizely to put a winning variation live for 100% of all traffic, even those visitor that were previously bucketed into another variation. Pausing a variation and changing traffic allocation in general will only affect future visitors. We're not altering the experience for those visitors that are already participating in the experiment. One of the reasons for this is that we're trying to keep the user experience consistent to avoid confusion for your visitors. However, we do recognize that this isn't very intuitive when it comes to the ‘Launch’ option on our new Results page beta, so we'll look into ways to improve this as we continue to improve our new Results page.
Optimizely
by Optimizely
‎06-02-2014 June 2, 2014
Status changed to: Great Idea!
 
Optimizely
by cubelodyte
‎12-11-2014 December 11, 2014

@Dodiet 

 

Thank you for your input. You make an excellent point (I'm not sure why I'm just now finding your response).

I expect that I will most likely follow your practice of creating what I call a HotFix Experiment based on the winning variation.

 

Of course, in order to ensure the new 100% experiment variation is not conflicting with any return visits to other variations in the original experiment, the original should be archived.

 

With that said, I'm left to wonder "What is the circumstance that would lead someone to use the Launch button at all?".

 

In fact, I'd be curious to know the percentage use it gets. 

 

I'm noticing that it appears on the results of Archived experiments. That could be dangerous if someone sets it. It could easily be forgotten.

Since the only way to view older, archived experiments is to unarchive them, someone could unwittingly be pushing old variations out simply by trying to examine the code of an older experiment. Granted, this would only occur for visitors who had already seen that variation, but if site code has changed in the interim, there would be limited potential for site breakage.

 

 

 

 

 

 

Level 6