Testing Haven

Statistical Power Calculator

What can your sample size actually detect?

Test parameters

%
%
MDE type
Test type(two-tailed is the safer default)

Your test is underpowered. With 10,000 visitors per variant, you only have 21.4% power to detect a 5.0% relative change — a real effect of this size would be missed 79% of the time. At this sample size, the smallest relative lift you can reliably detect (80% power) is 12.2%. Use the sample size calculator to plan a properly sized test.

Power vs. sample size

The pink line marks your current sample size; the green dashed line is the conventional 80% power target.

What this calculator does

This statistical power calculator answers the question every "not significant" test result should trigger: could this test even have detected the effect I care about? Enter your baseline conversion rate, the effect size that would matter to your business, and your per-variant sample size, and it returns the power of your test — the probability of detecting that effect if it is real — plus the smallest lift your sample can reliably detect at 80% power.

Use it before launch to sanity-check a planned test, or after a null result to decide whether "no significant difference" means "no effect" or just "not enough data".

The formula

For a two-proportion test with per-variant sample size nn, baseline rate p1p_1, and target rate p2p_2, the power of a two-tailed test at significance level α\alpha is:

1−β=Φ ⁣(∣p2−p1∣−z1−α/2 2pˉ(1−pˉ)np1(1−p1)+p2(1−p2)n)1 - \beta = \Phi\!\left( \frac{|p_2 - p_1| - z_{1-\alpha/2}\, \sqrt{\tfrac{2\bar{p}(1-\bar{p})}{n}} }{ \sqrt{\tfrac{p_1(1-p_1) + p_2(1-p_2)}{n}} } \right)

where pˉ\bar{p} is the average of the two rates, Φ\Phi is the standard normal CDF, and z1−α/2z_{1-\alpha/2} is the critical value (1.96 at 95% confidence). Intuitively: power is the chance that your observed difference clears the significance bar, computed under the assumption that the true effect equals your MDE.

A worked example

You ran a test with 10,000 visitors per variant on a 10% baseline, hoping for a 5% relative lift (10% → 10.5%). Plugging in gives power of roughly 21% — meaning that even if the variant truly delivered the hoped-for lift, this test would detect it only about one time in five. A null result from this test is nearly meaningless. To reach 80% power for that effect you would need about 57,800 visitors per variant (see the sample size calculator), or you could re-scope the test toward a bigger, more detectable change.

When to use it

  • Before launch: verify the planned sample size gives ≥80% power for the minimum worthwhile effect.
  • After a null result: distinguish "well-powered null — the effect probably is not there" from "underpowered null — we learned little".
  • When auditing someone else's readout: underpowering is the most common silent flaw in test claims.

Common mistakes

  • Computing post-hoc power from the observed effect. It is a disguised p-value and adds no information; always evaluate power against the effect size that matters, not the one you happened to observe.
  • Treating a null result from a 20%-power test as evidence of no effect. Absence of evidence is only evidence of absence when power is high.
  • Ignoring the winner's curse. Significant results from underpowered tests systematically overstate the true lift — budget for shrinkage when projecting impact.
  • Quietly relaxing alpha or switching to one-tailed after the fact to rescue an underpowered test — decide rigor settings before launch.

Frequently Asked Questions

What is statistical power in plain terms?

Power is the probability that your test detects an effect that is really there. A test with 80% power for a 5% lift will, if the true lift is exactly 5%, come back significant 80% of the time — and miss it the other 20%. Low power means real winners routinely die in your testing program while everyone blames the ideas.

What power level should I aim for?

80% is the widely accepted minimum; 90% is better when missing a true improvement is expensive. Below roughly 70%, a test is more of a coin flip than an experiment: not only do you miss most real effects, but the effects you do detect are systematically exaggerated (the "winner's curse"), because only the lucky overestimates clear the significance bar.

My test came back not significant. Does that prove there is no effect?

No. "Not significant" means the data are compatible with no effect — but if your power was low, the data are also compatible with a meaningful effect you simply could not see. Check this calculator: if your test only had 30% power for the effect size you care about, a null result tells you almost nothing. A well-powered null result, by contrast, is genuinely informative.

What is post-hoc power and should I compute it?

Post-hoc power plugs the observed effect from a finished test back into the power formula. It is misleading: it is just a transformation of your p-value and always looks bad for non-significant results. The useful question is prospective — "what power did my sample size give me for the effect size I care about?" — which is exactly what this calculator answers, using your MDE rather than the noisy observed effect.

How can I increase power without more traffic?

Four levers: (1) test bigger changes — power rises steeply with effect size; (2) use a more sensitive metric closer to the change, e.g. add-to-cart instead of purchase; (3) reduce metric variance with techniques like CUPED, which uses pre-experiment data (see our CUPED calculator); (4) accept a higher alpha or a one-tailed test — legitimate only when decided before launch.