Defining KPI's for E-commerce
I wanted to ask you all your opinion on defining the right KPI's when testing. I have some discussion on choosing the right KPI for e-commerce product pages. The test involves (really simple) changing the color of the 'put in basket' button.
The goal of this page is to lat people click on that button to actually enter the sales funnel. When testing button color you are trying to improve CTR's to the basket page and so forth.
My collegue says that we should measure conversion rate in terms of actual orders, while i say there is too much influence down the funnel; (basket, checkout pages) to actually say one color is performing better than the other. I think you should stick to the goal of a particular page and base your KPI's on that.
What are your thoughts on KPI definition?
CTR from the product page is of course an important metric but if you are employing a tactic to increase initial add to basket clicks but none of those extra visitors are actually purchasing then it is an un-effective change.
I would advise that you monitor both as KPIs. If you are an eCommerce company then your primary metric will have to be conversion rate.
Thank you for your response.
I think that RPV is a better overall primary metric combining conversion rate and average order value. In the end you want to grow revenue, even if your CR is sometimes negatively impacted. But that's looking at it overall..
For this test I ofcourse included both metrics. The problem is when I see an increase in CTR and a decrease in CR, I think concluding the test delivered no improvement is based on non-reliable data.
It's like standing in a book store, and the one book is really well presented and pursuades you to grab it from the shelf and put in your basket. When you walk up to the register there's a huge line with crying babies and smelly people. You estimate that you';ll have to wait 30 minutes to pay for the book so you decide to just lay it down somewhere and buy it at the bookstore 2 blocks further down where the cover is less attractive, but at least you're the first one in line to cash out. The other day someone else grabs the book with the original cover and the line is again that long, but he is on holiday and had more time. A factor influencing the process afterwards but not the persuasion at the shelves.
The steps between grabbing it from the the shelve and actually buying it has so many influence on your buying behaviour that it's not justifiable to say: the new cover didn't work. It did persuade customers to put the book in the basket but didn't convert because of the bad checkout procedure.
I think you should look at CR as an overall metric and optmize all those little steps on itself to improve it. For this test we could set an 'expected CR change rate' (i.e. 5% more or 5% less is allowed). If it exceeds that rating than you should be concluding that the test actually impacted conversionrate so far down the funnel and maybe stick to the original.
I agree that it's important to look at both metrics and weigh the results of each according to the hypothesis of your experiment. Although there are a lot of factors that could effect the actual purchase decision throughout the checkout process both of your variations would be exposed to those factors equally. Therefore, if you did see a marked improvement or decline in your final checkout metric it makes sense that it was caused by the different experience of the user higher in the funnel.
Identifying what rate of change in CR is required to determine that the experiences are in fact signficantly different is a great recommendation. Then, interpret the results together - if the CTR increases and CR is flat or increases than that is a win. If the CTR increases and the CR decreases based on the change rate you identified than that is not a win.
I've seen that by increasing the prominence, color or priority of calls to action higher in the funnel can signficantly increase engagement with those elements however, the people can be less qualified and therefore drop out of later steps at a greater rate. To keep going with your real life example - more people may pick up a book that is prominently displayed but when they get to the line to purchase they look at it again and think "hmmm, i came in here for a book about Optimization strategy and even though this Greek Isle Travel Guide looks amazing, I really don't need it!"