0 Likes

Automatic experiment duration estimator

Status: New
by George ‎08-08-2015 August 8, 2015

When setting up an experiment, it would be great to have an understanding of how long the experiment would need to run given the existing traffic to the point of conversion. Many folks use other systems (Mixpanel, Google Analytics, Omniture) to calculate their traffic to pages/screens and clicks/taps on buttons, etc.

 

Since these two systems are naturally separate, one is required to calculate this duration separately and manually.

 

It would be great if instead when setting up an experiment and goals, we enter the page the experiment will run on, and while setting up a goal, it could tell us how much traffic that button click, etc. gets and how long the experiment would need to run for in order to be statistically significant.

 

That would give us a much better sense of whether an experiment is going to be worth it or not. If we are able to see that an experiment is going to take 2 months to reach significance, I might spend more time on another experiment where I'd be able to learn the outcome more quickly.

Level 2

Comments
by
‎08-13-2015 August 13, 2015

It would be a good idea, but then it will be very hard to take into account what the average lifecycle is that you want a test to run in. If your usual conversion path length takes about a week than you want to make sure that you run your tests at least 1,2,3.. weeks exactly instead of switching off the test after 4 days although you might reach significance.

Level 10
by George
‎08-20-2015 August 20, 2015

I think these are calculations people are doing manually anyway. But, given existing data about how long it typically takes to complete a funnel being tested, I think this could be added to the calculation quite easily. I'm not sure you'd need to restrict your test lengths to the exact period of the typical conversion, mostly because as you launch experiments you don't have control over when people see the experiment vs. start a funnel. In my experience and research anyway, experiments have the best effect when performed on funnels that have a shorter completion period than a week.

Level 2
by
‎08-20-2015 August 20, 2015

Oh to clarify, I didn't mean that it should be the exact length of a conversion path but at least not less than that to make sure that you might not miss on conversion that take 8 days to happen from the first visit if you would only test for 7 days.

Level 10