Skip to content
TESTING HAVEN

IV · LEARN

Statistical Significance in A/B Testing, Explained

By Testing Haven · Published · Updated

"Is it significant?" is the most-asked and most-misunderstood question in A/B testing. This guide explains what statistical significance actually claims, what it doesn't, and the handful of mistakes that quietly turn "significant" results into fiction.

What a p-value actually says

The p-value answers one narrow question: if the variant truly changed nothing, how likely is data at least this extreme? A p-value of 0.03 means that a world with no real difference would produce a gap this large only 3% of the time. That makes the no-difference explanation implausible — which is all "significant" means.

Note what the p-value does not say. It is not the probability the variant is better (that is the Bayesian question — see the Bayesian calculator). It is not the size of the effect. And 1 − p is not your "chance of winning", which is why our calculator never reports 1 − p as a confidence figure. A tiny p-value with a huge sample can accompany a lift too small to matter; always read the confidence interval alongside it.

Significance level, confidence, and the two ways to be wrong

Before the test starts you pick a significance level, α: the false-positive rate you are willing to live with. The convention is α = 0.05, and "95% confidence" is the same setting described from the other side. A result is called significant when p < α.

That rule controls exactly one kind of error. Among tests where the variant truly changes nothing, at most 5% will be declared significant — a false positive, or Type I error. It says nothing about the opposite error, missing an effect that is really there (a false negative, or Type II error). That one is governed by statistical power, which is set by your sample size. Power is a planning decision: settle it with the sample size calculator before launch, because no amount of analysis can add it afterwards.

The machinery: the two-proportion z-test

For conversion metrics, significance is usually computed with a two-proportion z-test: the observed difference in rates divided by the standard error you would expect under the null hypothesis. The result (z) says how many standard errors apart the two groups are, and it is converted into the p-value using the normal distribution (our tour of probability distributions covers the bell curve if it is hazy). Our significance calculator runs this test, shows the confidence interval on the lift, and — for A/B/n tests — applies multiple-comparison corrections automatically.

The z-test leans on a large-sample approximation, and it flatters small tests. With 1 conversion out of 20 visitors against 6 out of 20, the z-test reports p = 0.037; the exact calculation gives p = 0.091. So when any expected cell count falls below 5, the calculator switches the p-value to Fisher's exact test and says so in the output.

A worked example

Control: 10,000 visitors, 1,000 conversions (10.0%). Variant: 10,000 visitors, 1,120 conversions (11.2%) — a 12% relative lift. The pooled rate is 10.6%, which gives a standard error of 0.435 percentage points if nothing were different. The observed gap of 1.2 points is z = 2.76 standard errors wide, and the two-tailed p-value is 0.006. That is below 0.05, so the result is significant at the 95% level; it would pass at 99% too.

Now the part most readouts skip. The 95% confidence interval on the relative lift runs from about +3.0% to +21.0% (in absolute terms, +0.35 to +2.05 percentage points). The variant is very likely better, but "12%" is only the middle of a wide range. Forecast from the low end of the interval, not from the point estimate.

One-tailed or two-tailed?

A two-tailed test asks whether the variant is different in either direction; a one-tailed test asks only whether it is better. In the example above the one-tailed p-value is half the two-tailed one — 0.003 instead of 0.006 — which is why one-tailed tests are tempting, and why switching to one after seeing the data is p-hacking. A one-tailed test is legitimate only if you chose it before launch and would take the same action whether the variant turns out flat or harmful. Its matching interval is one-sided as well: instead of a two-sided range, the calculator reports a 95% lower bound — here, a lift of at least +4.4%.

Significant is not the same as important

With enough traffic, trivial effects become significant. At one million visitors per arm, a move from 10.0% to 10.1% — a 1% relative lift — gives p = 0.019, with a 95% confidence interval on the lift of +0.2% to +1.8%. Real, and quite possibly too small to pay for the change. The reverse also holds: a non-significant result from a small test does not show the effect is zero, only that the test could not tell. Before declaring an idea dead, check what the test could have detected with the power calculator.

The three mistakes that fake significance

1. Peeking. Checking a running test and stopping the moment p < 0.05 inflates the 5% false-positive rate to about 14% with five looks, 19% with ten, and 25% with twenty. The fix is either a pre-committed sample size (use the sample size calculator) or an always-valid method (the sequential calculator). The full story is in the peeking problem.

2. Multiple comparisons. Four variants, five metrics, three segments = 60 chances for a fluke. Even a plain four-variant test, uncorrected, carries up to an 18.5% chance of at least one false positive. Corrections like Holm's method keep the family-wise error rate honest: three variants with raw p-values of 0.004, 0.03 and 0.04 look like three wins, and after Holm's correction (0.012, 0.06 and 0.06) only the first survives. Our significance calculator applies Holm by default for multi-variant tests, with Bonferroni as the stricter option.

3. Broken randomization. If the traffic split itself is off — a sample ratio mismatch — the p-value is answering a question about corrupted data. The significance calculator checks the split automatically on the visitor counts you enter, and the standalone SRM checker is there for checking a split on its own. The SRM guide explains what to do when the check fails.

Reading a significant result like an adult

A significant result earns three follow-up questions. Was the test adequately powered (an underpowered win exaggerates the true effect — check the power calculator)? Does the confidence interval include effect sizes you would not ship? And does the result survive the data-quality checks? If yes to all three, ship it — and expect the measured lift to shrink a little in production. That is not failure; that is regression to the mean.

A pre-readout checklist

  • The sample size and stopping rule were fixed before launch, or you used an always-valid method.
  • The traffic split passes the SRM check.
  • The primary metric and the test type (one- or two-tailed) were chosen in advance.
  • With more than one variant, the p-values are corrected.
  • You have read the confidence interval, and its low end is still worth shipping.
  • A null result has been checked against power before the idea is written off.

For the other way to read the same data, see Bayesian vs frequentist. And if you are not sure which tool fits where you are in a test, answer two quick questions and get pointed at the right one.

  1. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value does and does not tell you.
  2. Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70.The correction for multiple comparisons.
  3. Campbell, I. (2007). Chi-squared and Fisher–Irwin Tests of Two-by-Two Tables with Small Sample Recommendations. Statistics in Medicine, 26(19), 3661–3675.Choosing between the chi-squared and exact tests for small samples.
  4. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B Tests: Why It Matters, and What to Do About It. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17), 1517–1525.How peeking inflates false positives.
  5. Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.Why a significant result from a small test overstates the effect.
  6. Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19), 2156–2164.Sample ratio mismatch as a data-quality check.