Rich-text Reply

What is a better success metric than "do no harm"?

jtabert 04-14-16

What is a better success metric than "do no harm"?



My client has been wanting to run a lot of tests where we for instance add a purchase CTA on an eBook download page where there was previously no purchase path. We tend to fall back to a "do no harm" to the primary CTA on the page as our main success metric.


Does anyone have any ideas on other kinds of success metrics to use when you're adding a new path or feature to a page? If you have an existing purchase link it would be easy to test a button treatment and use clickthrough on the CTA as a success metric. But if there was no link like that in the first place, things get more muddied. 


Any thoughts would be much appreciated!



Level 1

JasonDahlin 04-15-16

Re: What is a better success metric than "do no harm"?

Is the primary KPI also a "purchase" link?

If so, I'd look at total purchases rather than click through.  Or look at the total number of click-throughs (clicks on the original plus clicks on the new).


Ultimately, the website has a goal (or purpose) of some sort (usually selling stuff, but it could be email signups or simply pages viewed if you are a non-profit informational site).  That goal should be one of the metrics that are evaluated with every experiment.


Click throughs on a landing page are nice when testing the effectiveness of getting people into the purchase funnel, but ultimately, you want to make sure you are attacting people who are wanting to purchase, so "orders" should be a KPI for every experiment. (It may not be the primary KPI, but it should at least be referenced).


--Jason Dahlin
Analytics and Testing Guru Smiley Happy

Experimentation Hero
robertchan 04-15-16

Re: What is a better success metric than "do no harm"?

One thing I am almost done building involves determining a user's idle time on the page. After determining the amount of idle time the average idler reaches, some type of interstitial is delivered to the user. In your case, you could consider delivering a modal to get the user to purchase something and measure conversions off the modal. To accomplish all of this, you can implement a variety of such tests using Optimizely's custom events. I would also consider making use of cookies and checking when the best time is to deliver appropriate content to users who say fall into x category of purchases based on their cookie history.
Robert Chan

Experimentation Hero