TESTING HAVEN

II · ANALYZE

A/B Test Report

Every check on one test, on one page

An A/B test report puts every check on one test in one place: whether the traffic split is sound, whether the difference is significant, whether the test was big enough, and how large the lift really is. For example, 10.0% against 11.2% on 10,000 visitors per group is significant (p = 0.0058), but the lift to plan on is +6.6%, not +12.0%.

The test

How it was run

Confidence
Test type
%
%
%

Enter visitors and conversions for both groups to see the full report.

How the mathematics works

What this report does

Reading an A/B test properly takes more than one number. This page runs the checks an experienced analyst would run, in the order they matter, and turns them into a verdict and a paragraph you can paste into a write-up. Enter the visitors and conversions for control and one variant, and say how the test was meant to run.

The five checks

  1. Is the data sound? A chi-square test compares the visitors in each group with the split you planned. A p-value under 0.001 signals a sample ratio mismatch: the groups are no longer comparable and the report stops there. The SRM checker does the same test on its own.
  2. Is the difference significant? A two-proportion z-test, switching to Fisher's exact test for very small samples, exactly as in the A/B test calculator.
  3. Was the test big enough? The power the test had to detect the lift you were looking for, at the sample sizes it actually reached. At least 80% is adequate; under 50%, the test would miss a real lift of that size more often than it found it.
  4. How big is it really? The observed lift, shrunk toward zero to allow for the fact that results which reach significance are the lucky ones. This is the calculation in the winner's curse calculator, and it uses your belief about how large real lifts usually are.
  5. How sure can you be? The probability that the variant is truly better than control, given the same belief. The Bayesian reading with no prior belief is shown beside it for comparison.

How the verdict is reached

  • Do not act on this test: the traffic split fails its check.
  • A credible win: significant, and at least a 95% chance the variant is truly better once large lifts are treated as rare.
  • A win on paper, but not yet a safe one: significant, but short of that 95%.
  • The variant is worse than control: a significant drop.
  • No effect of the size you were looking for: not significant, from a test with at least 80% power for your target lift.
  • Inconclusive: not significant, from a test too small to tell.

A worked example

A test sends 10,000 visitors to each group. Control converts 10.0% and the variant 11.2%: an observed lift of +12.0% with p = 0.0058. The split is exactly as planned, so the data is sound, and the result is significant. The test was small for a 5% target, with only 21% power, which is part of why the observed lift should not be taken at face value: the lift to plan on is +6.6%. Even so, the chance the variant is truly better is 97.3%, so the verdict is a credible win.

Change the variant to 10.89% and the picture shifts. The result is still significant (p = 0.040), but the chance the variant is truly better falls to 93%, and the verdict becomes a win on paper that is not yet safe.

What the report cannot see

  • Peeking. If the test was stopped the first time it looked significant, its p-value is too optimistic. Use the sequential testing calculator for tests you check as they run.
  • Several metrics or segments. If this is the best of many comparisons, the result is weaker than it looks.
  • Novelty. A lift that fades after the first days will not show in totals.
  • Whether wins like this are usually real. That depends on how often your ideas work, which is the subject of the false positive risk calculator.
What does the report check?

Five things, in the order they matter. First, whether the traffic split matches what you planned: if it does not, nothing else can be trusted. Second, whether the difference is statistically significant. Third, whether the test was large enough to detect the lift you cared about. Fourth, how large the true lift is likely to be once you allow for the winner’s curse. Fifth, how sure you can be that the variant is really better.

Why is my significant result called "not yet a safe win"?

Because significance and safety are different bars. A result that only just clears p < 0.05 leaves real doubt once you allow for how rare large lifts are. In the example on this page, a result with p = 0.040 is statistically significant, yet the chance the variant is truly better is 93%, short of the 95% the report asks for. Repeating the test is the usual remedy.

Why does it ask what lift the test was meant to detect?

To judge whether the test was big enough. Power is always power to detect something. A test can be far too small to detect a 2% lift and perfectly adequate for a 20% one. Enter the smallest lift that would have mattered to you; if you planned the test with a minimum detectable effect, use that.

What is the difference between "inconclusive" and "no effect"?

Both mean the result was not significant. "No effect of the size you were looking for" means the test had at least 80% power to detect your target lift and did not find it, so the absence of a result is informative. "Inconclusive" means the test was too small for that: a real effect could easily have been missed, and the test has not settled the question.

Can I use it for a test with several variants?

The report covers one variant against control. For a test with several variants, run the A/B test calculator first: it corrects for the extra comparisons and tells you which variant the result is really about. The link from there opens this report for that variant.

Where do my numbers go?

The calculations run in your browser. The numbers and the test name are kept in the page’s address, which is what lets you bookmark the report or share it as a link. The site’s analytics record which page was viewed and never the address after the question mark.

  1. Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19), 2156–2164.Why a sample ratio mismatch invalidates a test, and how to check for one.
  2. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value does and does not tell you.
  3. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).Why experiments with low statistical power cannot be trusted.
  4. Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.How far significant results overstate the true effect, and how that depends on power.
  5. van Zwet, E. W., & Cator, E. A. (2021). The Significance Filter, the Winner’s Curse and the Need to Shrink. Statistica Neerlandica, 75(4), 437–452.Why selecting on significance inflates estimates, and the case for shrinkage.