Skip to content
TESTING HAVEN

IV · LEARN

A/B Testing Glossary

Every term the calculators use, in plain language

70 terms from A/B testing statistics, each defined in two or three plain sentences and linked to the calculator or guide that puts it to work. Every term has its own address, so you can link straight to a definition.

A

A/A test

A test in which both groups get exactly the same experience. Any difference it reports is noise or a fault in the setup, so it is used to check that a testing tool splits traffic and counts conversions correctly.

At 95% confidence about 1 A/A test in 20 will look significant by chance alone. Many more than that points to a fault.

Always-valid p-value

Also called anytime-valid p-value

A p-value built to stay trustworthy however often you look at it. You can check it after every visitor and stop the moment it falls below your threshold without pushing the false positive rate above alpha.

The price is data: it needs more visitors to reach significance than an ordinary p-value read once at the end.

B

Bayesian A/B testing

An approach that describes what is known about each version’s true rate as a probability distribution, updated by the data. It answers direct questions such as “what is the probability the variant is better?” and “how much could I lose by shipping it?”

Its answers depend on the prior, and it is not immune to peeking: stopping the moment a probability looks good still inflates mistakes.

Bonferroni correction

The simplest fix for multiple comparisons: divide alpha by the number of comparisons and hold each one to that stricter threshold. With 5 comparisons at 95% confidence, each needs a p-value below 0.01.

It guarantees the family-wise error rate but is cautious, so real effects are missed more often. The Holm correction gives the same guarantee and never finds less.

C

Confidence interval

A range of plausible values for the true effect, worked out from the data. A 95% confidence interval comes from a method that captures the true value in 95% of tests. If the interval for the difference excludes zero, the result is statistically significant at that level.

Its width is as informative as its position: a wide interval means the test has not pinned the effect down, whatever the p-value says.

Credible interval

The Bayesian counterpart of a confidence interval: a range that contains the true value with a stated probability, given the data and the prior. A 95% credible interval can be read the way people wish they could read a confidence interval: there is a 95% probability the true value lies inside it.

CUPED

Also called Controlled-experiment Using Pre-Experiment Data

A technique that uses each visitor’s behavior from before the test to remove predictable variation from the metric. The answer is the same but less noisy, so a test reaches the same power with fewer visitors.

The saving grows with how well past behavior predicts present behavior. It helps most for returning visitors and does nothing for people with no history.

Related:
Variance · Sample size · Power

E

F

False positive risk

Also called FPR

Of the tests that come out significant, the share where the change truly does nothing. It depends on how often your ideas work as well as on alpha and power, and it is usually much higher than alpha.

If 1 idea in 10 works, a two-tailed win at 95% confidence and 80% power has a 22% chance of being a false positive, not 5%.

G

H

Holm correction

Also called Holm–Bonferroni method

A step-by-step version of Bonferroni. Sort the p-values, hold the smallest to alpha divided by the number of comparisons, the next to alpha divided by one fewer, and stop at the first that fails.

It controls the family-wise error rate exactly as Bonferroni does and never rejects fewer results, so there is no reason to prefer plain Bonferroni.

L

M

Mann–Whitney U test

Also called Wilcoxon rank-sum test

A rank-based test that compares two groups without assuming their values follow a bell curve. It is useful for revenue and other skewed metrics as a check on the t-test.

It asks whether one group tends to have larger values, which is not the same question as whether the averages differ.

N

O

P

P-value

The probability of seeing a difference at least as large as the one you measured if the change truly made no difference. A small p-value means the data would be surprising under “no difference”.

It is not the probability that the result is due to chance, and it is not the probability that the variant is better.

Peeking

Also called optional stopping, repeated significance testing

Checking a fixed-horizon test before it has finished and stopping as soon as it looks significant. Each extra look is another chance for noise to cross the line, so the real false positive rate climbs well above alpha.

With 5 evenly spaced looks at 95% confidence, a change that does nothing is declared a winner 14.2% of the time. Looking is harmless; stopping because of what you saw is what does the damage.

R

S

Statistical significance

A result is statistically significant when its p-value is below alpha: the data would be unlikely if the change did nothing. It is evidence that a difference exists, not proof, and it says nothing about whether the difference is large enough to matter.

T

U

V

W

Z

  1. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.The standard reference for the vocabulary of online experiments, including Twyman’s law.
  2. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value is and is not.
  3. Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.Type S (sign) and Type M (magnitude) errors.
  4. Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.The false discovery rate and the procedure that controls it.
  5. Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70.The step-down correction for multiple comparisons.
  6. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B Tests: Why It Matters, and What to Do About It. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17), 1517–1525.Peeking, and always-valid p-values.
  7. Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM ’13), 123–132.CUPED: using pre-experiment data to reduce variance.