How to Size an A/B Test (Sample Size, MDE, and Power)

Every A/B test implicitly bets that its sample size is large enough to see the effect it is hunting. Most losing bets are placed before launch. This guide walks through sizing a test properly: choosing an MDE, setting power, computing the sample, and converting it into a runtime.

Step 1: choose the minimum detectable effect

The MDE is a business decision disguised as a statistical one: the smallest lift that justifies shipping. Compute it backwards from value — if a 2% relative lift on this funnel pays for the work, your MDE is 2%, and your test must be sensitive enough to see 2%. Sizing for the lift you hope for (say 15%) produces a small test that will end "not significant" against the 4% lift you actually got.

Step 2: set alpha and power

Significance level α = 0.05 (95% confidence) and power = 80% are the standard defaults. Raise confidence to 99% for risky, hard-to-reverse changes; raise power to 90% when missing a genuine winner is expensive. Both make the test bigger — power from 80% to 90% costs about a third more traffic.

Step 3: compute the sample size

Plug baseline rate, MDE, α, and power into the sample size calculator. The output is per variant — an A/B/C test needs it three times over. The driving math: required n scales with 1/MDE², so halving the MDE quadruples the sample. This is why "let's also detect tiny effects" is an expensive sentence.

Step 4: convert to a runtime

Divide the required sample by your weekly test-eligible traffic — or better, use the MDE & duration calculator, which shows the detectable effect week by week. Round up to whole weeks to keep the weekday/weekend mix balanced. If the answer is "14 weeks", do not negotiate with the math: pick a bolder change, a more sensitive metric, or variance reduction with CUPED.

Then honor the contract

The computed sample size is only meaningful if you actually run to it. Stopping at the first significant flicker voids the error guarantees (see the peeking problem). When the test ends, verify the traffic split with the SRM checker, then read the result with the significance calculator. If it comes back null, check what your test could actually detect with the power calculator before declaring the idea dead.