IV · LEARN
The Pocket Ledger
Fifteen checks for a test you can trust, on one printed page
Most bad A/B test results come from a handful of avoidable mistakes. These 15 checks cover them, in the order they come up. Tick them off here, or print the page and keep it by your desk: it fits on one sheet.
Before you start
Choosing the metric after the test is choosing the answer.
It is a business decision, not a forecast.
A number fixed in advance cannot be argued with later.
Weekday and weekend visitors behave differently.
While it runs
A broken split is cheapest to find early.
Stop at the first of 5 looks that crosses the line, and 14% of changes that do nothing become winners.
A test that changed halfway is two half-tests.
When it ends
If the split is wrong, no other number can be trusted.
Significant at 95% does not mean 95% likely to be real.
Winning tests overstate their lift.
Tests that lost or showed nothing are part of the record.
How to use it
The list is short on purpose. Each line is a decision that is easy to make before a test and almost impossible to make honestly afterwards: which metric counts, how many visitors you need, when you are allowed to look. Settle those first and most of the statistics takes care of itself.
If you only adopt three, make them these: fix the sample size before you start, do not stop early because the result looks good, and check the traffic split before you read anything else. The A/B test report runs the checks in the last section for you, and the glossary explains any term here that is new.
Sources
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.The standard practitioner’s reference for running tests that can be trusted.
- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19), 2156–2164.Why the traffic split is checked before anything else.
- Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society: Series A (General), 132(2), 235–244.What stopping at the first significant look does to the false positive rate.
- Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).Why significant at 95% does not mean 95% likely to be real.
- Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.Why winning tests overstate their lift.