TESTING HAVEN

I · PLAN

Sample Size Calculator for Revenue Metrics

How many visitors does a revenue test need?

A test on revenue per visitor usually needs more traffic than a test on conversion rate, because the size of each order adds its own noise. With an average of 3.10 per visitor and a standard deviation of 18, detecting a 10% change at 95% confidence and 80% power takes 52,925 visitors per group.

Your metric

%
Change is
Confidence
%
Test type

You need 52,925 visitors per group (105,850 total for an even A/B test) to detect a change in the average from 3.10 to 3.41 with 95% confidence and 80% power.

Why revenue needs so many visitors

Your metric's standard deviation is 5.8 times its average. Most visitors spend nothing and a few spend a lot, so one visitor says very little about the average. The sample size grows with the square of that ratio.

Halving the change you want to detect needs four times the visitors: 211,700 per group for a change of 0.16.

How the mathematics works

What this calculator does

It tells you how many visitors each group needs for a test on an average: revenue per visitor, average order value, pages per session. Enter the metric's average and standard deviation, and the smallest change that would matter, and it returns the sample size per group.

The formula

For a metric with standard deviation σ\sigma and a difference to detect δ\delta, each group needs

n=2 σ2 (z1−α/2+z1−β)2δ2n = \frac{2\,\sigma^2\,(z_{1-\alpha/2} + z_{1-\beta})^2}{\delta^2}

visitors, where z1−α/2z_{1-\alpha/2} is the critical value for the confidence level (1.96 at 95%) and z1−βz_{1-\beta} the one for the power (0.84 at 80%). With more than one variant, α\alpha is divided by the number of variants, matching the correction applied when the test is analysed.

Individual visitors' spending is nowhere near normally distributed, and that does not matter here. The test compares averages over thousands of visitors, and those averages are close to normal even when the underlying amounts are heavily skewed.

A worked example

A shop earns 3.10 per visitor on average, with a standard deviation of 18: the spread is 5.8 times the average, which is typical when most visitors buy nothing. To detect a 10% change (0.31) at 95% confidence and 80% power, each group needs 52,925 visitors. Detecting a 5% change needs four times as many: 211,700.

Where does a spread like that come from? Suppose 3% of visitors buy, spending 103 on average, with a standard deviation of 80 between orders. Revenue per visitor then averages 3.10 with a standard deviation of about 22, and a 10% change needs 82,119 visitors per group. A 10% lift in the 3% conversion rate alone needs 53,211. The revenue test needs 1.5 times the traffic, because order values add their own noise on top of the yes-or-no of buying. That is why many programs decide on conversion and treat revenue as a check.

Getting more from the same traffic

  • Reduce the noise with pre-experiment data. Returning visitors' past spending predicts their future spending. The CUPED calculator shows how much that saves.
  • Cap extreme values. A handful of very large orders can account for much of the variance. Capping at, say, the 99th percentile is a common, defensible choice if it is made before the test.
  • Decide on a less noisy metric. Conversion rate usually reaches a verdict sooner; analyse revenue alongside it with the revenue per visitor calculator.
Why can I not use the ordinary sample size calculator for revenue?

The ordinary calculator is for conversion rates, where every visitor is a yes or a no and the noise follows from the rate alone. Revenue per visitor is an amount: most visitors contribute nothing and a few contribute a great deal. Its noise depends on how spread out those amounts are, which the standard deviation measures, and that has to be supplied separately.

Where do I find the standard deviation?

From visitor-level data: one row per visitor with what they spent, zeros included. A spreadsheet's STDEV function or your analytics tool's export gives it directly. Be careful not to use the standard deviation of order values among buyers only; that leaves out the zeros and understates the average and the spread relative to it.

What if I only know the conversion rate and the average order value?

You can build the per-visitor figures from them. If a share p of visitors buy, with average order value m and standard deviation of order values s, then the average per visitor is p × m and the variance per visitor is p × (s² + m²) − (p × m)². Take the square root of the variance for the standard deviation to enter here.

How can I run a revenue test with less traffic?

Three ways. Test a bigger change, since the sample size falls with the square of the difference. Cap extreme orders at a high percentile before analysis, which cuts the standard deviation sharply at a small cost in bias. Or use CUPED, which removes the part of each visitor's spending that their earlier behaviour already predicts.

Does this work for other metrics?

Yes, for any metric that is an average of per-visitor numbers: pages per session, time on site, items per order, support tickets per account. Enter that metric's average and standard deviation. It does not suit ratios of two totals, such as revenue per session when visitors have several sessions, which need a different variance.

  1. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. 2nd edition. Routledge.Sample size and power for a difference between two means.
  2. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.Variance of revenue metrics, capping, and sample size in online experiments.
  3. Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM ’13), 123–132.Reducing the variance of a metric with pre-experiment data (CUPED).