A/B Testing Guides

Deeper explanations behind the calculators — the statistics, the failure modes, and how to run experiments people can trust.

Statistical Significance in A/B Testing, Explained

What p-values, confidence levels, and significance actually mean in an A/B test — and the mistakes that make "significant" results wrong.

A practical walkthrough of sample size calculations: choosing an MDE, setting power, and translating the answer into test duration.

How Bayesian probability-to-be-best and expected loss compare to p-values and confidence intervals, and when each approach fits.

Why repeatedly checking a fixed-horizon test inflates false positives, with simulations, and how sequential methods like mSPRT fix it.

How to detect sample ratio mismatch, the most common causes, and why you should throw away results from a test that fails an SRM check.