Rich-text Reply
Highlighted

Auto-shutoff for experiments based on metrics?

optimizelyrose 02-07-19
Accepted Solution

Auto-shutoff for experiments based on metrics?

The problem I'm facing is that my team often runs experiments overnight or over the weekend. We currently manually stop any experiments whose variations are performing too poorly, but there's not always someone monitoring performance of an experiment. Does Optimizely have some built-in functionality (maybe via the dashboard) for marking an experiment or variations to automatically be stopped as soon as a variation has lower conversion metrics than the control and has an acceptable threshold of confidence?

JasonDahlin 02-08-19
 

Re: Auto-shutoff for experiments based on metrics?

You would have to build it yourself to pull the current test results via the API and send an API request to stop the experiment that meets the thresholds you've indicated.
--Jason Dahlin
Analytics and Testing Guru Smiley Happy


Experimentation Hero

Re: Auto-shutoff for experiments based on metrics?

Thank you for your reply, Jason! Yes, it seems we'll have to build it if no faster solution exists yet. Thanks again!
optijon 02-12-19
 

Re: Auto-shutoff for experiments based on metrics?

@optimizelyrose my company has actually been thinking of building a tool to solve this problem. It's a common problem and one that I think wastes a lot of time. Send me a message if you'd be up for a quick chat about what would be most useful to you.

Level 1
JasonDahlin 02-13-19
 

Re: Auto-shutoff for experiments based on metrics?

@optijon,

 

I'm curious about what environment you work in where turning off experiments at night or over the weekend is useful and productive. Are you A/B testing time-sensitive landing pages (like a headline on a blog that is only viral for 2 days) or similar?  Is there a minimum significance level that you use when deciding to turn it off early (like, "50% confident in a negative 5% KPI")?  On such tests, are you typically finding bugs in the experiment or are they just "really really bad ideas"?

 

For reference, typically our experiments have 300k+ unique visitors per month and need to run at least 4-6 weeks before seeing statistical significance. Once or twice a year we run an experiment that tanks in the first 2-3 days that we stop, find what might be going wrong and then relaunch a new version of the experiment after the issues are fixed, but we would never turn one off at night then turn it back on the next day. This is also why we launch experiments on Tuesdays so that we have 4 work days to find any issues in case we do need to turn it off before the weekend.

Tuesday = launch at midnight, verify tracking is working when we come into the office that morning, run a quick look that afternoon to make sure it didn't break the site Man LOL

Wednesday = monitor

Thursday = 2 full days of data, discuss any concerns

Friday = 3 full days of data, if experiment needs to be cancelled, this is when it would happen

 

The only time we have ever cancelled an experiment after the first Friday is if there is a code release that "breaks" it (e.g., the element we are modifying no longer exists/works)

--Jason Dahlin
Analytics and Testing Guru Smiley Happy


Experimentation Hero