TESTING HAVEN

I · PLAN

A/B Test Sample Size Calculator

How many visitors does your test need?

The sample size an A/B test needs depends on your baseline conversion rate and the smallest lift you want to detect. To detect a 5% relative lift on a 10% baseline at 95% confidence and 80% power, you need 57,763 visitors per group. Halving the lift you are looking for roughly quadruples that number.

Test parameters

%
%
MDE type(a 5% lift on 10% → 10.50%)
%
Test type(two-tailed is the safer default)

You need 57,763 visitors per group (115,526 total for an even A/B test) to detect a change from 10.00% to 10.50% with 95% confidence and 80% power. With fewer visitors, a real effect of this size would go undetected more than 20% of the time. To translate this into a runtime, use the test duration calculator.

Sample size vs. minimum detectable effect

Halving the MDE roughly quadruples the required sample size (note the log scale). The rust-colored dot marks your current inputs.

How the mathematics works

What this calculator does

This sample size calculator tells you how many visitors each variant of your A/B test needs before you start. You give it four numbers — your baseline conversion rate, the minimum detectable effect (MDE) you care about, your significance level, and your desired statistical power — and it returns the required sample size per variant, with a chart showing how that requirement explodes as the MDE shrinks.

Sizing the test up front is the single highest-leverage habit in experimentation: it prevents both the underpowered test that wastes four weeks to conclude nothing, and the endless test that gets stopped the day it flickers significant.

The formula

For a two-proportion test comparing baseline rate p1p_1 against target rate p2p_2 (baseline + MDE), the required sample size per variant is:

n=(z1−α/22pˉ(1−pˉ)+z1−βp1(1−p1)+p2(1−p2))2(p2−p1)2n = \frac{\left( z_{1-\alpha/2} \sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta} \sqrt{p_1(1-p_1) + p_2(1-p_2)} \right)^2}{(p_2 - p_1)^2}

where pˉ=(p1+p2)/2\bar{p} = (p_1 + p_2)/2 is the average rate, z1−α/2z_{1-\alpha/2} is the normal quantile for your significance level (1.96 at 95%, two-tailed), and z1−βz_{1-\beta} is the quantile for your power (0.84 at 80%). The denominator is the squared effect — which is why halving the MDE roughly quadruples the required sample.

With more than one variant, α\alpha is divided by the number of variants before the quantile is taken (a Bonferroni adjustment). Each variant is compared against control, so this keeps the plan consistent with the multiple-comparison correction applied when the test is analyzed.

A worked example

Baseline conversion rate 10%, and you want to detect a 5% relative lift (10% → 10.5%) at 95% significance and 80% power, two-tailed. Plugging in: p1=0.10p_1 = 0.10, p2=0.105p_2 = 0.105, pˉ=0.1025\bar{p} = 0.1025. The formula gives 57,763 visitors per variant, or about 115,500 total. If your site gets 20,000 visitors a week into the test, that is a six-week runtime — knowledge worth having before you build the variant, not after. (Check the MDE & duration calculator to see what is detectable at your actual traffic.)

How to choose each input

  • Baseline rate: pull the last 2–4 weeks of the exact metric and audience your test will use. A stale or mismeasured baseline skews everything.
  • MDE: work backwards from business value — the smallest lift that pays for the engineering time. Anything smaller than that is not worth detecting.
  • Significance: 95% is standard; use 99% for risky, hard-to-reverse changes.
  • Power: 80% is standard; 90% for decisions where missing a real winner is costly.

Common mistakes

  • Stopping early at the first significant reading. The sample size is the contract that makes your error rates real. Breaking it by peeking voids the warranty — use the sequential testing calculator if you need early stopping.
  • Relative/absolute MDE confusion. A "5% lift" on a 10% baseline could mean 10.5% or 15% depending on interpretation: 57,763 visitors per variant versus 686, a difference of about 84x. This calculator labels both explicitly.
  • Sizing for the hoped-for effect instead of the minimum worthwhile effect. Optimism makes tests small and inconclusive.
  • Ignoring extra variants. An A/B/C test needs the quoted n in each of the three groups, and a stricter significance level for each comparison. Set the number of variants and the calculator accounts for both.
  • Not running full weeks. Round your runtime up to whole weeks so weekday/weekend mix does not bias the sample.
Why do I need to calculate sample size before starting a test?

Because the alternative is running until the result "looks done", which is how false positives are manufactured. Fixing the sample size in advance guarantees your stated error rates actually hold: your significance level controls false positives, and your power controls false negatives. It also tells you before you invest whether the test is feasible at all at your traffic level.

What is a reasonable minimum detectable effect (MDE)?

The MDE should be the smallest lift that would still be worth shipping — not the lift you hope to get. For most mature conversion funnels, real UI changes move metrics by 1–10% relative, so MDEs in the 2–10% range are typical. Picking an unrealistically large MDE (like 25%) gives you a small, fast test that will almost certainly end "not significant" because real effects that big are rare.

What do 80% power and 95% significance actually mean?

95% significance (alpha = 0.05) means that if the variant truly changes nothing, you have at most a 5% chance of a false alarm. 80% power (beta = 0.2) means that if the true effect is exactly your MDE, you have an 80% chance of detecting it — and a 20% chance of missing it. These are the conventional defaults; raise power to 90% for important decisions if you can afford the roughly 34% larger sample.

What is the difference between relative and absolute MDE?

A relative MDE is a percentage of your baseline: a 10% relative lift on a 5% conversion rate means moving to 5.5%. An absolute MDE is in percentage points: a 1-point absolute lift on 5% means moving to 6%. Confusing the two changes sample sizes enormously — a 1-point absolute lift on a 5% baseline is a 20% relative change. This calculator supports both; the toggle shows you the implied target rate either way.

Can I stop the test early if it reaches significance before the planned sample size?

Not with a fixed-horizon test — stopping at the first significant reading is peeking and inflates your false positive rate well beyond 5%. If you want the option to stop early, use a sequential testing method (see our sequential testing calculator), which charges a slightly larger sample in exchange for always-valid results.

Why might my number differ slightly from other calculators?

Sample size tools use slightly different textbook formulas. This one uses the pooled-variance two-proportion formula with exact normal quantiles and no continuity correction. statsmodels, which works from an arcsine effect size, lands within about 0.2% of it (57,756 versus 57,763 for a 10% baseline and a 5% relative lift). Tools that take the null variance from the baseline rate alone give somewhat smaller numbers, and tools that add a continuity correction give larger ones. Differences of a few percent are normal; a difference of 2x or more usually means one tool is one-tailed, or is reading the MDE as absolute instead of relative.

  1. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.Planning sample size, power and minimum detectable effect.
  2. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).Why experiments with low statistical power cannot be trusted.