What this report does
Reading an A/B test properly takes more than one number. This page runs the checks an experienced analyst would run, in the order they matter, and turns them into a verdict and a paragraph you can paste into a write-up. Enter the visitors and conversions for control and one variant, and say how the test was meant to run.
The five checks
- Is the data sound? A chi-square test compares the visitors in each group with the split you planned. A p-value under 0.001 signals a sample ratio mismatch: the groups are no longer comparable and the report stops there. The SRM checker does the same test on its own.
- Is the difference significant? A two-proportion z-test, switching to Fisher's exact test for very small samples, exactly as in the A/B test calculator.
- Was the test big enough? The power the test had to detect the lift you were looking for, at the sample sizes it actually reached. At least 80% is adequate; under 50%, the test would miss a real lift of that size more often than it found it.
- How big is it really? The observed lift, shrunk toward zero to allow for the fact that results which reach significance are the lucky ones. This is the calculation in the winner's curse calculator, and it uses your belief about how large real lifts usually are.
- How sure can you be? The probability that the variant is truly better than control, given the same belief. The Bayesian reading with no prior belief is shown beside it for comparison.
How the verdict is reached
- Do not act on this test: the traffic split fails its check.
- A credible win: significant, and at least a 95% chance the variant is truly better once large lifts are treated as rare.
- A win on paper, but not yet a safe one: significant, but short of that 95%.
- The variant is worse than control: a significant drop.
- No effect of the size you were looking for: not significant, from a test with at least 80% power for your target lift.
- Inconclusive: not significant, from a test too small to tell.
A worked example
A test sends 10,000 visitors to each group. Control converts 10.0% and the variant 11.2%: an observed lift of +12.0% with p = 0.0058. The split is exactly as planned, so the data is sound, and the result is significant. The test was small for a 5% target, with only 21% power, which is part of why the observed lift should not be taken at face value: the lift to plan on is +6.6%. Even so, the chance the variant is truly better is 97.3%, so the verdict is a credible win.
Change the variant to 10.89% and the picture shifts. The result is still significant (p = 0.040), but the chance the variant is truly better falls to 93%, and the verdict becomes a win on paper that is not yet safe.
What the report cannot see
- Peeking. If the test was stopped the first time it looked significant, its p-value is too optimistic. Use the sequential testing calculator for tests you check as they run.
- Several metrics or segments. If this is the best of many comparisons, the result is weaker than it looks.
- Novelty. A lift that fades after the first days will not show in totals.
- Whether wins like this are usually real. That depends on how often your ideas work, which is the subject of the false positive risk calculator.