Skip to content
TESTING HAVEN

IV · LEARN

How to Size an A/B Test (Sample Size, MDE, and Power)

By Testing Haven · Published · Updated

Every A/B test implicitly bets that its sample size is large enough to see the effect it is hunting. Most losing bets are placed before launch. This guide walks through sizing a test properly: choosing an MDE, setting power, computing the sample, and converting it into a runtime.

Step 1: choose the minimum detectable effect

The MDE is a business decision disguised as a statistical one: the smallest lift that justifies shipping. Compute it backwards from value — if a 2% relative lift on this funnel pays for the work, your MDE is 2%, and your test must be sensitive enough to see 2%. Sizing for the lift you hope for (say 15%) produces a small test that will end "not significant" against the 4% lift you actually got.

Say whether you mean relative or absolute, every time. On a 10% baseline, a 5% relative lift is 10% → 10.5% and needs 57,763 visitors per variant. A 5-point absolute lift is 10% → 15% and needs 686. Same words in the meeting, an 84-fold difference in traffic.

Step 2: set alpha and power

Significance level α = 0.05 (95% confidence) and power = 80% are the standard defaults. Alpha is the chance of a false alarm when nothing changed; power is the chance of catching the MDE when it is really there. Raise confidence to 99% for risky, hard-to-reverse changes; raise power to 90% when missing a genuine winner is expensive. Both make the test bigger. For the 5% lift above, going from 80% to 90% power takes the requirement from 57,763 to 77,328 visitors per variant — about a third more — and going from 95% to 99% confidence takes it to 85,950, about half as much again. A one-tailed test needs 45,500, which is 21% fewer, but it only counts if you committed to it before launch.

Step 3: compute the sample size

Plug baseline rate, MDE, α, and power into the sample size calculator. The driving math: required n scales with 1/MDE², so halving the MDE roughly quadruples the sample. This is why "let's also detect tiny effects" is an expensive sentence. In words, the formula weighs the noise in a conversion rate against the squared size of the effect you want to see, then scales the result up for the confidence and power you asked for. At a 10% baseline with the standard settings:

Relative MDERates comparedVisitors per variant
20%10% → 12%3,841
10%10% → 11%14,751
5%10% → 10.5%57,763
2.5%10% → 10.25%228,555
1%10% → 10.1%1,419,073

The baseline matters as much as the MDE. Rare events are noisier relative to their size, so a 10% relative lift needs 14,751 visitors per variant at a 10% baseline, 53,211 at 3%, and 80,682 at 2%. If your primary metric converts at 2%, ask whether an earlier step in the funnel would answer the same question with a fraction of the traffic.

Step 4: account for extra variants

The output is per variant, so an A/B/C test needs it three times over. Extra variants also raise the bar for each comparison, because the analysis corrects for multiple comparisons. The planning calculators take the number of variants and size the test with a Bonferroni-adjusted α, so the plan matches what the significance calculator will do at readout. For the 5% lift example: one variant against control needs 57,763 per arm (115,526 in total); two variants need 69,952 per arm (209,856 in total); three need 77,047 per arm (308,188 in total). Every extra variant should earn its place.

Step 5: convert to a runtime

Divide the required sample by your weekly test-eligible traffic — or better, use the MDE & duration calculator, which shows the detectable effect week by week. Suppose 20,000 eligible visitors a week, split evenly: that is 10,000 per arm per week, so 57,763 takes 5.8 weeks. Round up to six full weeks to keep the weekday/weekend mix balanced.

Read the same numbers the other way round and you see what your traffic can detect: a 12.2% lift after one week, 8.6% after two, 6.0% after four, 4.9% after six, and 3.5% after twelve. The returns diminish quickly, which is why "let it run a bit longer" rarely rescues an insensitive test.

Uneven splits cost traffic. Sending 10% of visitors to the variant instead of half needs about 2.8 times the total traffic for the same sensitivity; a 70/30 split needs about 19% more. Use an uneven split to limit exposure to a risky change, not to go faster.

What an undersized test costs

Suppose you skip the arithmetic and run 10,000 visitors per variant, hoping for that 5% lift. The test has 21% power: a real 5% improvement comes back "not significant" almost four times out of five, and a good idea gets shelved. The wins that do clear the bar are the lucky overestimates, so the lift you report is inflated — the winner's curse. Check any planned or finished test with the power calculator.

When the answer is too long

If the answer is "14 weeks", do not negotiate with the math. Change the test:

  • Test a bolder change. A 10% effect needs about a quarter of the traffic a 5% effect does.
  • Use a more sensitive metric. Add-to-cart moves more, and more often, than purchase.
  • Reduce variance. With CUPED, a pre-experiment covariate correlated at 0.5 with your metric removes 25% of the required sample; at 0.7 it removes 49%.
  • Widen the audience. More eligible pages or segments means more visitors per week.
  • Monitor sequentially. If the effect may be large, the sequential calculator lets you stop as soon as the evidence is strong, at the cost of a longer test when it is marginal.

Then honor the contract

The computed sample size is only meaningful if you actually run to it. Stopping at the first significant flicker voids the error guarantees (see the peeking problem). When the test ends, verify the traffic split with the SRM checker, then read the result with the significance calculator. If it comes back null, check what your test could actually detect with the power calculator before declaring the idea dead.

Common sizing mistakes

  • Counting the wrong traffic. Only visitors who reach the tested experience count. If the change is on step three of checkout, size from step-three visitors, not site-wide sessions.
  • Using a stale baseline. A rate measured last quarter, or on a different audience, shifts the answer. A baseline of 8% instead of 10% changes the requirement by more than a quarter.
  • Reading per-variant as total. The calculator quotes visitors for each arm.
  • Sizing for the hoped-for lift. Optimism makes tests small and inconclusive.
  • Adding variants for free. Each one needs its own arm and tightens the threshold for all of them.

Sizing checklist

  1. Baseline taken from the last 2–4 weeks of the same metric and audience.
  2. MDE set from business value, and stated as relative or absolute.
  3. Alpha, power, test type and number of variants fixed before launch.
  4. Runtime rounded up to whole weeks, using test-eligible traffic rather than site-wide sessions.
  5. Stopping rule written down: run to the planned sample, or monitor with an always-valid method.

For what the result means once the test is done, read statistical significance, explained.

  1. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.Choosing a minimum detectable effect, power and duration.
  2. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).Why experiments with low statistical power cannot be trusted.
  3. Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM ’13), 123–132.CUPED, for shortening tests with pre-experiment data.