Total revenue - using the Optimizely calculator
We use the Optimizely calculator (https://www.optimizely.com/resources/sample-size-calculator) when pondering on how long to keep running a test. The total revenue is somewhat of a different type of goal - is the sample size calculator working as well on this type of goal?
Solved! Go to Solution.
You're right to consider Revenue as a different type of goal!
The short answer is that the sample size calulator is not designed to work with Revenue goals. Try using a conversion goal closely tied to revenue, for example completed purchase, as your goal for estimating sample size.
The longer answer:
The Sample Size Calculator is designed to function for binary goals - goals that have a conversion rate. (In this context, 'binary' just means that we're tracking whether or not a user triggered a given goal condition; the conversion rate is % of visitors that triggered the goal / all visitors). Revenue, as a continous variable, does not have a conversion rate, and would require a custom calculation based on the variance of your site's average revenue/visitor calculation in order to calculate sample size.
An example of a binary goal that would work for estimating sample size would be completed purchase; use the baseline percentage of visitors that normally complete a purchase as the baseline conversion rate % input in the calculator.
Would you find it useful to use a purchase conversion rate as your experiment's primary goal? (You can still report on revenue later, but would be using the conversion rate to ultimately declare whether a test variation has won or lost.)
This approach is extremely common; in addition to the statistical benefit, in most cases, the behavior you're actively seeking to influence is whether or not they complete a purchase, rather than the size of purchase, in which case purchase conversion rate is a more precise KPI.
Try plugging in some of your historical conversion rate values into the calculator.
Thanks Hudson, that was a clever work-around! We messure completed purchases anyway so I think one single calculator is well enough for us.