What this calculator does
This calculator answers the question that comes after a winning test: how much of this lift will I keep? Enter your result, and the range of lifts you would have considered plausible before the test, and it returns the lift you should expect after launch, the share of the observed lift that is likely to survive, and a table showing how much a test of your size overstates true lifts of different sizes.
Why winners are overstated
An observed lift is the true lift plus noise. A test becomes a winner when its observed lift clears a bar, and the results that clear a bar are disproportionately the ones where the noise ran in their favour. Each test is unbiased on its own. The collection of tests you decide to ship is not.
The effect is strongest when a test is small for the lift it is chasing. In the worked example below, the test cannot call any lift under 9.0% significant. If the true lift were 3%, it would come out significant only 10% of the time, and in those cases it would report 11.2% on average: 3.7 times the truth.
The calculation
The calculator combines two things: the observed lift with its standard error , and your belief before the test, expressed as a normal distribution centred on no lift with standard deviation . The expected true lift is then the observed lift multiplied by a factor between 0 and 1:
The noisier the test (large ) and the more surprising a big lift would have been (small ), the more the estimate is pulled toward zero. You enter the belief as a range: "most real lifts fall within ±10%" means 95% of them, which sets . The uncertainty that remains has standard deviation .
This kind of estimate has a useful property: it does not need a further correction for having been selected. An estimate that already accounts for what was plausible beforehand stays valid whether or not you only look at it because the test was significant.
A worked example
A test with 10,000 visitors per group converts 10.0% of control and 11.2% of the variant: an observed lift of +12.0% with a standard error of 4.6%. If you believed beforehand that most real lifts for this kind of change fall within ±10%, the expected true lift is +6.6%, about 55% of what was observed, with a 95% chance that it lies between -0.1% to +13.3%. If you believed most real lifts fall within ±5%, the expected true lift is +2.8%, about 23% of the observed figure.
Neither answer says the test was wrong. Both say that a +12.0% reading from a test this size is partly luck, and how much depends on how common large lifts really are.
How much tests overstate, by power
Leaving beliefs aside, the average overstatement among significant results depends only on the test's power to detect the true lift (for a two-tailed test at 95% confidence):
- 90% power: 1.06×
- 80% power: 1.12×
- 50% power: 1.4×
- 20% power: 2.3×
- 10% power: 3.7×
A well-powered test barely overstates. A badly underpowered one reports several times the truth whenever it reports anything, which is why a striking lift from a small test deserves the most suspicion. Check your design with the power calculator before launch.
What it assumes
- A sceptical starting point. Before the test, a change like this was as likely to hurt as to help, and large lifts in either direction were rare. If you have good reason to expect an improvement, the result is conservative.
- Normal approximations. The observed relative lift is treated as normally distributed around the true lift, with the same standard error the A/B test calculator uses for its confidence interval. This is reliable for the sample sizes of a typical conversion test and rough for very small ones.
- One result. If the variant shown is the best of several, or the metric is the best of several, the overstatement is larger than shown.
Common mistakes
- Forecasting revenue from the raw lift of a winner. Use the expected true lift for projections.
- Adding up the raw lifts of every shipped test. Each is overstated, so the total is too, often by a wide margin.
- Trusting the biggest lifts most. A very large lift from a modest sample is the result most likely to be mostly noise.
- Confusing this with whether the win is real. That is a separate question, answered by the false positive risk calculator.