What this calculator does
It tells you how many visitors each group needs for a test on an average: revenue per visitor, average order value, pages per session. Enter the metric's average and standard deviation, and the smallest change that would matter, and it returns the sample size per group.
The formula
For a metric with standard deviation and a difference to detect , each group needs
visitors, where is the critical value for the confidence level (1.96 at 95%) and the one for the power (0.84 at 80%). With more than one variant, is divided by the number of variants, matching the correction applied when the test is analysed.
Individual visitors' spending is nowhere near normally distributed, and that does not matter here. The test compares averages over thousands of visitors, and those averages are close to normal even when the underlying amounts are heavily skewed.
A worked example
A shop earns 3.10 per visitor on average, with a standard deviation of 18: the spread is 5.8 times the average, which is typical when most visitors buy nothing. To detect a 10% change (0.31) at 95% confidence and 80% power, each group needs 52,925 visitors. Detecting a 5% change needs four times as many: 211,700.
Where does a spread like that come from? Suppose 3% of visitors buy, spending 103 on average, with a standard deviation of 80 between orders. Revenue per visitor then averages 3.10 with a standard deviation of about 22, and a 10% change needs 82,119 visitors per group. A 10% lift in the 3% conversion rate alone needs 53,211. The revenue test needs 1.5 times the traffic, because order values add their own noise on top of the yes-or-no of buying. That is why many programs decide on conversion and treat revenue as a check.
Getting more from the same traffic
- Reduce the noise with pre-experiment data. Returning visitors' past spending predicts their future spending. The CUPED calculator shows how much that saves.
- Cap extreme values. A handful of very large orders can account for much of the variance. Capping at, say, the 99th percentile is a common, defensible choice if it is made before the test.
- Decide on a less noisy metric. Conversion rate usually reaches a verdict sooner; analyse revenue alongside it with the revenue per visitor calculator.