Email Notifications for experiments

Status: Done
by siherron Level 2 ‎08-28-2014 August 28, 2014

Would it be possible to trigger a dynamic notification (lets go with email for simplicity sakes) for experiments that:

 

  • Have hit statistical significance for specified set of goals (so 1 or more)
  • Have hit a consistent statistical significance over a user specified duration

In both instances the test designer would dictate the duration and the specific goals of interest.

 

Business Case.

Imagine running 20+ tests across multiple accounts and having to check them periodically to understand whether the experiments have completed based upon multiple goals, some primary, some secondary. Like you would in a marketing agency, for example.

 

Perhaps another approach would be to allow users to set a scheduled/periodic email containing a CSV or Excel report for a specific experiment. That might save users time on exports and, dare I say, an easy (consistent) interim solution for those of us who like to scrape the results data out of our experiments and push it into "fancy" dashboards.

 

Just sayin' - Thoughts?

 

Smiley Happy

Status: Done
Level 2

Comments
by Optimizely
‎09-04-2014 September 4, 2014 - edited ‎09-04-2014 September 4, 2014

This is a great idea @siherron 

 

This is something that will be addressed as we launch our upcoming results API! This API will allow you to programatically recieve results data. You can then also Programmatically start or stop experiments at this time. 

 

However, you should be careful about polling results until the first time they hit significance. 

 

Evan Miller explains why its important to sent a sample size for your test up front and not "continuously monitor" p values (Chance to Beat Baseline)

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

 

You can get access to our Sample Size Calculator here: 

https://www.optimizely.com/resources/sample-size-calculator

 

Best,

Darwish 

Optimizely
by Optimizely
‎09-04-2014 September 4, 2014
Status changed to: Done
 
Optimizely
by Optimizely
‎09-11-2014 September 11, 2014 - edited ‎09-11-2014 September 11, 2014

@siherron - This is now available. Check out the references below on how to programmatically retrieve restults with the Optimizely Results API. 

 

 

Blog Post: http://blog.optimizely.com/2014/09/11/programmatically-retrieve-results-with-optimizelys-rest-api/

Documentationhttp://developers.optimizely.com/rest/#get-results

Sample Code: http://developers.optimizely.com/samples/#getting-results-via-api-13

 

Optimizely
by siherron Level 2
‎09-12-2014 September 12, 2014

Aces - I saw this yesterday, when brushing up for the Dev Certificate. Thanks for the note.

 

Si

Level 2
by tFox
‎10-08-2014 October 8, 2014 - edited ‎06-24-2016 June 24, 2016

@siherron ,

 

Check this out. Its automated email reports for cases just like yours: 

 

http://trevorfox.com/2014/10/automate-optimizely-reports-results-api

 

You had great ideas- let me know how this works out for you.

Level 2