What this calculator does
This calculator answers the question that follows every winning A/B test: how likely is it that this win is real? Enter the share of your tested ideas that really work, the confidence level and power of your tests, and it returns the false positive risk: the probability that a statistically significant win comes from a change with no effect at all.
The answer is usually larger than people expect. If one idea in ten works, a significant win at 95% confidence and 80% power has a 22.0% chance of being a false positive, not 5%.
To get a feel for it, play Real or noise?: ten tests that all reached significance, and you guess which wins are real.
Why it is not 5%
A 95% confidence level means that a change with no effect will look significant 5% of the time. That is a statement about changes with no effect. It says nothing about how many such changes you test. If most of your ideas do nothing, most of your tests are chances for a false win, and the few real wins have to share the stage with them.
A medical screening test works the same way. A test that is wrong for only 5% of healthy people still produces mostly false alarms if the condition is rare, because there are so many healthy people to be wrong about.
The formula
Let be the share of tested ideas that really work, the power, and the chance that an idea with no effect shows a significant win. Bayes' rule gives:
For a two-tailed test, : a no-effect change is equally likely to look significantly better or significantly worse, and only the first is shipped as a win. For a one-tailed test, .
If an independent repeat of the test also wins, both terms are squared: against . Two false wins in a row are far less likely than two real ones, which is why replication is so effective.
A worked example
Suppose 10% of the ideas you test really improve the metric, and you run two-tailed tests at 95% confidence with 80% power. Out of 1,000 tests, 100 are of ideas that work, and 80 of those are detected. The other 900 are of ideas with no effect, and 2.5% of them, 22.5 tests, come out as significant wins anyway. So of 102.5 significant wins, 22.5 are false: 22.0%. The chance a given win is real is 78.0%.
If you repeat a winning test and it wins again, the risk falls to 0.9%. To reach a 5% risk from a single test instead, you would need p < 0.0094, which takes more visitors (see the sample size calculator). A program where one idea in three works starts from a much better place: 5.9%.
Finding your success rate
The success rate is the one input you have to estimate. The best source is your own record: the share of finished tests that produced a significant win on the metric they were designed to move. That observed share contains some false wins and misses some real ones, and the two roughly cancel. If is your observed share of wins, the implied share of ideas that work is . With 12% of tests winning, that gives about 12%.
Published figures from large experimentation programs run from about 8% to 33%, and the calculator shows the risk at each of those. If you have no record yet, treat the result as a range rather than a number.
What it assumes
- Two kinds of idea. Each idea either has no effect or improves the metric by the amount the test was powered for. Real effects come in all sizes; smaller ones are detected less often, which raises the risk. Ideas that make things worse are counted as having no effect, which slightly overstates it.
- A clean test. The sample size was fixed in advance, nobody stopped early on a good day, there is one primary metric, and the traffic split is healthy. Peeking (see the sequential testing calculator), several metrics, and sample ratio mismatch (see the SRM checker) all push the real false positive rate above the nominal one.
- Your estimate of the success rate. The result is only as good as that input, so try more than one value.
How this differs from the Bayesian reading
The A/B test calculator reports a Bayesian probability that the variant beats control. By default that uses a flat prior, which treats every conversion rate as equally likely before the data arrive and takes no account of your track record. False positive risk does the opposite: it ignores the size of the observed difference and uses only the fact that the result was significant, together with how often your ideas work. The two answer different questions, and a result is on firmest ground when both are reassuring.
Common mistakes
- Reading 95% confidence as a 95% chance the variant is better. Confidence describes how the test behaves when nothing is going on, not how likely your result is to be true.
- Reading the p-value as the false positive risk. A p-value of 0.03 does not mean a 3% chance the win is false.
- Trusting wins from underpowered tests. Low power leaves false wins as a larger share of what comes out significant (check yours with the power calculator).
- Shipping a surprising, high-stakes win without repeating it. A second independent win is the cheapest way to cut the risk.