Rich-text Reply

Service to detect changes to underlying code

CROmetrics 08-03-15

Service to detect changes to underlying code

Is anyone aware of a service to detect changes in the underlying code when an experiment is running?  One common problem I have seen is that a deploy can break an experiment, and the only way to know that is to look at it.  CEOs have a way of finding this stuff at 5pm on a Friday.  Having an experiment break is generally very expensive given the investment that goes into a well-executed experiment.


If no one is aware, then I might build it.  Let me know if you're interested.

Re: Service to detect changes to underlying code

Hi @CROmetrics,

 

This is a great and insightful question.  We usually try to coach our customers on organizational visibility into the testing queue and production releases so there is transparency when an experiment is running on a page that is now changing.

 

I am not aware of an existing service that can programmatically check for underlying page code that disagrees with client-side jQuery.  Some cursory internet research reveals that a very basic check that could be performed is for the simple presence of elements likely to change, using the .length method.

 

More robust approaches utilize the DOMNodeInserted or DOMSubtreeModified methods.

 

Essentially, you could then use these methods to build out an asynchronous polling script that probes for changes in the DOM as compared to the variation code run by Optimizely (i.e. compare the DOM results to the js_component value of the /variations endpoint in our REST API).  If the results of each check don't match, or some Optimizely jQuery function returns null, show an alert if the force variation parameter or test cookie is present, or if the IP is within the corporate network.  Or send some XHRs to a listening service that sends emails or pages developers.

 

We'd love to hear from the Community if anyone has (tried to) tackle(d) this before as well!

Harrison Krat
Solutions Architect | Optimizely, Inc.
harrison@optimizely.com
 
tobyurff 08-05-15
 

Re: Service to detect changes to underlying code

Hi Chris!

There don't seem to be any pre-built tools specifically for this purpose, but there are quite a few that allow you to monitor for changes on a page and receive notifications based on this. You could use those as part of your workflow.

Siebe from Blue Mango Interactive, one of our Solutions Partners, has blogged about this and their workflow here: http://geek.bluemangointeractive.com/how-we-monitor-our-ab-tests-with-pingdom/

Do you think this could be useful for your day-to-day work?

Toby
Optimizely
tomfuertes 08-05-15
 

Re: Service to detect changes to underlying code

Hi @CROmetrics!

We use GhostInspector for this purpose. Their screenshot and diff tools are great if you scope them to the parent of the elements being modified. Tests can run every 15 minutes and the platform is surprisingly stable (basically a marketing friendly GUI to selenium). Typically we point the instance to an optimizely_x control and optimizely_x variation preview link. Add a couple jQuery('original').text()==='control thing we expect' and/or JSON.stringify(.css())=== '{float:right}'. The screenshots are also very helpful for regressions and have helped us several times when we were moving say a suggested products module from the category pages to the home page and then that CSS got ripped out and/or made more modular. We were notified and allowed to fix it in < 30m (without having to wait for the data to drop!).

TLDR: Huge fan of GhostInspector. Very powerful, fast, and affordable. We tried a lot of automated tools ourselves and found the delta in features to not be worth the maintenance and setup.
____
- Tom Fuertes | CTO @ CROmetrics / LinkedIn
"Most Impactful Use of Personalization" and "Experience of the Year" Optie award winner.


Experimentation Hero