Rich-text Reply

Concluding Experiment

RPruitt 06-23-14

Concluding Experiment

If have hit the "Launch" button to distribute the winning variation to my site, do I need to pause the experiement? I didn't see a "stop" or "complete" option.

 

Thanks!

Level 2

Finge 06-23-14
 

Re: Concluding Experiment

Hi RPruitt! As soon as you hit launch and confirm all the traffic will be directed to your winning variation. The only way Optimizely can serve up anything else then what is live on your own site (original) you will need to keep the experiment going.
Level 2
Amanda 06-24-14
 

Re: Concluding Experiment

Yep, @Finge  is correct! If you click "launch", your experiment will still be running, but 100% of the traffic will be directed to the specified variation. 

 

However, please keep in mind that this only applies to *new* visitors. All the visitors who were intially bucketed into one of the other variations will continue to see the same variation. To get around this, we reommend that you duplicate the experiment, pause the original experiment, and start the new experiment with 100% traffic allocation set to the winning variation. This will ensure all visitors (both new and returning) will see the winning variation. 

 

Let me know if you have any questions.  Also, I would love to hear more about the success of your test since it sounds like you found a winner. Would you be willing to tell us about it on our testing successes board

Optimizely
Finge 06-24-14
 

Re: Concluding Experiment

Thanks for the clarification,  @AmandaS ! So basically for every new experiment where you want to build on the learnings from the previous experiment - create a new one?

Level 2
RPruitt 06-24-14
 

Re: Concluding Experiment

Thanks for the responses! I feel like it might be best to just make the site wide change to website code instead. Thoughts on that vs creating a new campaign again?

Level 2
Finge 06-24-14
 

Re: Concluding Experiment

Depends on how much has been tested and improved. If you still feel like you have more that is worth testing, and the implementation takes some time/resources I would perhaps do more testing first before implementing. 

 

Either way what you decide I would keep the winning variation in rotation, and create a new challenger with other elements that could improve even more, while implementing.

 

Perhaps you can share a URL? I'd be happy to suggest elements to test.

Level 2
RPruitt 06-24-14
 

Re: Concluding Experiment

[ Edited ]

Sure, we tested the current text "Apply Online" vs "Get a Quote" which we ran for 14 days. We saw a 7.6% increase over original Apply Online. I launched winning test yesterday.

 

Here is an example of the button placement:

 

http://www.proliability.com/professional-liability-insurance/nurses

Level 2
Finge 06-24-14
 

Re: Concluding Experiment

Great. I don't know your product well, but here's a thought. Change the headline to "Get peace of mind with Liability Insurance". Working with messaging on a page is more likely to give you bigger increase in conversion rates.

Another thing, but not likely one that will give dramatic results, is to add » to the "Get a Quote" button. I would even try "Get peace of mind »" on the button ;-)
Level 2
Amanda 06-24-14
 

Re: Concluding Experiment

@Finge - to you previous question: I definitely recommend creating a new experiment where you want to build on learnings from previous experiments. This will preserve the accuracy of the results and make sure visitors are seeing the variations you expect.

Also, I love the suggestion about adding a » to the CTA. I have seen this work very well in the past as well.

Let me know if I can help in any other way.
Optimizely
laurenm 09-25-14
 

Re: Concluding Experiment

Is there a way to permanently modify the code of the site for the winning variation through optimizely? Or would we need to subscribe to optimizely forever to continue to have the winning variation to show?

 

Thanks!

Level 2
KerriA 09-25-14
 

Re: Concluding Experiment

[ Edited ]

Thanks for this great thread.  I was trying to figure out if 'launching' the winning variation would stop the experiment and thus preserve my results but I see that is not the case.

 

@Amanda is this something that is on Optimizely's roadmap?  It would be great to launch a winning variation but keep the results as they were at that moment in time for future reference.

 

Duplicating the experiment and pausing the existing one seems onerous especially when you have a dozen experiments running that conceivably could reach statistical significance simultaneously.

 

Thanks!

Level 2
Amanda 09-29-14
 

Re: Concluding Experiment

Hey @KerriA , Yes we are looking into different ways to make launching a winning variation as easy as possible. I would also recommend that you add this suggestion (and include your complatin about duplicating an experiment being onerous) to the product ideas board. Our Product Team reviews this board frequently and uses the ideas in their relase planning. 

 

In the meantime, you could always launch the winning variation, and then change the date filter on your results page to end on the day you made this traffic allocation switch. That way, you could still see the results at that moment in time for future reference. 

Optimizely
Amanda 09-29-14
 

Re: Concluding Experiment

@laurenm  - Yes, you can make the changes permanent on your site without having to keep your experiment running. To do this, a developer will need to adjust the code on your site to match the changes you made via your experiment. Setting your variation to 100% is a stopgap measure until your team can make the changes directly within the code. Let me know if you have any questions on this. 

Optimizely
laurenm 09-29-14
 

Re: Concluding Experiment

Thanks @Amanda - That's good to know! Is there a way to find the code that we made during the experiement? Then, I'm guessing it's more of a copy/paste game for the developer.

Level 2
Amanda 09-29-14
 

Re: Concluding Experiment

@laurenm - you can find the code that you used in the experiment by clicking the "Edit code" button in the bottom right of your variation in the editor.

 

However, this code is jQuery, a languaged used to dynamically modify elements on your page. Rather than copy pasting this jQuery to your site, your developer will actually want to modify the HTML itself to match the varaition. If you send your dev team the variation code (as mentioned above), they should easily be able to determine how they can adjust the site code to match the changes made via your experiment. Be sure to tell them which set of pages need to be changed as well. 

 

Another way to see the code is to view the running variation on your site. Use 'right click' to view the source of your page and you'll be able to see the udpated page code. 

Optimizely
Ed 12-15-14
 

Re: Concluding Experiment

@Amanda this thread was very helpful to me as I was confused by the Launch Winner button. There is no change to the UI after confirming you want to do this so I was unsure if any change had taken place and kept hitting it thinking there must have been an error in my browswer that was not allowing it to happen. In addition as the experiment still runs, there is no alert on the homepage stating that the experiment was concluded with all traffic going through the winning variation. Definitely a confusing communication issue that your team should probably addresss.

Ed
Level 2