Calculating stat. significance for revenue and the choice of using RPV over AOV
Where could I find a more detailed description how statistical significance for the Total revenue goal is being calculated?
Why is Revenue per visitor the chosen metric over AOV, which for most businesses is much higher valued KPI?
Is it just becuase Optie's revenue tracking script doesn't record purchase counts?
Thanks in advance,
RE: Revenue Statistical Significance - See this note from our resident statistician, Leo: (I know its titled RPV but is relevant for total revenue)
We're also planning on publishing a white paper on the statistics of calculating significance for continuous variable metrics like Revenue.
How does Stats Engine work with a revenue per visitor goal?
Stats Engine works as intended on revenue per visitor goals. You can look at your results at any time you want and get an accurate assessment of your error rates on winners and losers, as well as confidence intervals on the average revenue per visitor.
In fact, your estimates should be more reliable, sooner, because we now correct for the inherent skewness in calculations based on revenue. One of the changes we made with Stats Engine is we now compute skew corrected test statistics for revenue (or any other goal that can potentially take on many values). Significance values are adjusted by a correction factor which estimates skewness from your currently running experiment. Not only does this make results reliable with considerably fewer visitors, but it also results in a more powerful test (when looking at historical revenue tests, the number of conclusive results jumped by a factor of 1.5).
Another feature of skew corrections is the resulting confidence intervals are no longer symmetric, but naturally adapt in the direction of the skew.
RE: RPV over AOV -
I can't speak to whether RPV or AOV is a more valuable metric; they accomplish different things, and for some customers I've spoken to, RPV is exactly what they've been looking for.
If you'd like to use Optimizely to report on AOV, my colleague @Brett offered the following advice:
A couple things to consider when selecting AOV or Revenue Per Visitor/User/Customer:
- Your comparison assumes that all observations areindependent and identically distributed (iid). This means each observation shares nothing in common other than being drawn from the same distribution. Here's an example of why this might not hold true when using AOV: It is a reasonable assumption that a company will have repeat orders from customers. If today a customer purchases a video game console it is also a reasonable assumption that sometime in the future they might purchase accessories (like a controller), video games, etc. As such, when accounting for AOV you have to be selective, such as only using a customer's first AOV for analysis, excluding any subsequent order's AOV (from same customer). The problem here is, this is valuable information! Obviously you'd want to know the full impact of how an experiment/variation influenced purchase behavior. This is often why Revenue Per Visitor/User/Customer is used.
- On Revenue Per Visitor/User/Customer, it is important that you allow the same data collection period for all visitors to ensure you collect all data required for the comparison. As an example, if you were to end an experiment after four weeks’ time, visitors who entered the experiment Week 1 would have had four full weeks of purchase history tied to them, whereas visitors who entered the experiment Week 4 would only have a single weeks’ worth of purchase history tied to them. To account for this, if you chose to stop your experiment after Week 4, you'd want to data collection to continue through Week 8 to allow equal time for RPV to build for all individuals included in the experiment. In this example, you would not count in your analysis any new visitors to the experiment after Week 4, and not attribute any transaction data from Week 5 and beyond to customer who entered your experiment Week 1 (or from Week 6 and beyond to a customer who entered in Week 2, and so on).
You mentioned that "We're also planning on publishing a white paper on the statistics of calculating significance for continuous variable metrics like Revenue." Wonder when will this paper be ready?
I am trying to compare a change on a paper which may affect how often a visitor will click on certain links. A visitor may click on those links multiple types, and so it is not a conversion.
In this case, if I want to rely on the statistical significance values reported the Results page, I can only use the Revenue goal, is that right?
Unfortunately we haven't published any specific publication re: statistics for revenue. I highly recommend exploring the academic paper published by our Optimizely Stats Engine designers.
There's also some similar commentary to what we've been discussing in the Optimizely Knowledge Base article on using statistics with Optimizely.
I think your question mostly is meant to figure out not the statistics, but simply tracking events for continous variables. You can actually accomplish that today by toggling the results page to 'totals'. You can read how here.
Sorry it took me so long to get back to you!