TESTING HAVEN

III · DIAGNOSE

Winner’s Curse Calculator

How much of your lift will survive?

Your test result

Input
%
Confidence

Enter visitors and conversions for both groups to see how much of the lift to expect to keep.

How the mathematics works

What this calculator does

This calculator answers the question that comes after a winning test: how much of this lift will I keep? Enter your result, and the range of lifts you would have considered plausible before the test, and it returns the lift you should expect after launch, the share of the observed lift that is likely to survive, and a table showing how much a test of your size overstates true lifts of different sizes.

Why winners are overstated

An observed lift is the true lift plus noise. A test becomes a winner when its observed lift clears a bar, and the results that clear a bar are disproportionately the ones where the noise ran in their favour. Each test is unbiased on its own. The collection of tests you decide to ship is not.

The effect is strongest when a test is small for the lift it is chasing. In the worked example below, the test cannot call any lift under 9.0% significant. If the true lift were 3%, it would come out significant only 10% of the time, and in those cases it would report 11.2% on average: 3.7 times the truth.

The calculation

The calculator combines two things: the observed lift d^\hat{d} with its standard error ss, and your belief before the test, expressed as a normal distribution centred on no lift with standard deviation τ\tau. The expected true lift is then the observed lift multiplied by a factor between 0 and 1:

E[ d∣d^ ]=d^⋅τ2τ2+s2E[\,d \mid \hat{d}\,] = \hat{d} \cdot \frac{\tau^2}{\tau^2 + s^2}

The noisier the test (large ss) and the more surprising a big lift would have been (small τ\tau), the more the estimate is pulled toward zero. You enter the belief as a range: "most real lifts fall within ±10%" means 95% of them, which sets τ=10%/1.96\tau = 10\% / 1.96. The uncertainty that remains has standard deviation s τ2/(τ2+s2)s\,\sqrt{\tau^2 / (\tau^2 + s^2)}.

This kind of estimate has a useful property: it does not need a further correction for having been selected. An estimate that already accounts for what was plausible beforehand stays valid whether or not you only look at it because the test was significant.

A worked example

A test with 10,000 visitors per group converts 10.0% of control and 11.2% of the variant: an observed lift of +12.0% with a standard error of 4.6%. If you believed beforehand that most real lifts for this kind of change fall within ±10%, the expected true lift is +6.6%, about 55% of what was observed, with a 95% chance that it lies between -0.1% to +13.3%. If you believed most real lifts fall within ±5%, the expected true lift is +2.8%, about 23% of the observed figure.

Neither answer says the test was wrong. Both say that a +12.0% reading from a test this size is partly luck, and how much depends on how common large lifts really are.

How much tests overstate, by power

Leaving beliefs aside, the average overstatement among significant results depends only on the test's power to detect the true lift (for a two-tailed test at 95% confidence):

  • 90% power: 1.06×
  • 80% power: 1.12×
  • 50% power: 1.4×
  • 20% power: 2.3×
  • 10% power: 3.7×

A well-powered test barely overstates. A badly underpowered one reports several times the truth whenever it reports anything, which is why a striking lift from a small test deserves the most suspicion. Check your design with the power calculator before launch.

What it assumes

  • A sceptical starting point. Before the test, a change like this was as likely to hurt as to help, and large lifts in either direction were rare. If you have good reason to expect an improvement, the result is conservative.
  • Normal approximations. The observed relative lift is treated as normally distributed around the true lift, with the same standard error the A/B test calculator uses for its confidence interval. This is reliable for the sample sizes of a typical conversion test and rough for very small ones.
  • One result. If the variant shown is the best of several, or the metric is the best of several, the overstatement is larger than shown.

Common mistakes

  • Forecasting revenue from the raw lift of a winner. Use the expected true lift for projections.
  • Adding up the raw lifts of every shipped test. Each is overstated, so the total is too, often by a wide margin.
  • Trusting the biggest lifts most. A very large lift from a modest sample is the result most likely to be mostly noise.
  • Confusing this with whether the win is real. That is a separate question, answered by the false positive risk calculator.
What is the winner's curse in A/B testing?

It is the tendency of winning tests to overstate their own effect. A test is declared a winner when its observed lift clears a significance bar, and observed lifts are the true lift plus noise. The results that clear the bar are disproportionately the ones where the noise happened to push upward, so on average a winner's observed lift is larger than its true lift. The name comes from auctions, where the winning bidder is the one who overestimated the item's value the most.

My test was run correctly. Why would the lift still be overstated?

Because the overstatement comes from selection, not from a mistake. In the example on this page, a test with 10,000 visitors per group can only call a lift significant if it observes one beyond ±9.0%. If the true lift is 3%, the test is significant only 10% of the time, and when it is, the average reported lift is 11.2%: 3.7 times the truth. Every individual test is unbiased; the set of tests you choose to ship is not.

How do I choose the range of plausible lifts?

Think of the changes you have tested before, not the one in front of you. If almost all of them moved the metric by less than 10% in either direction, enter 10. Past tests are the best guide: look at the lifts that held up after launch or in repeat tests, not the raw lifts of the winners, which are themselves inflated. If you are unsure, the calculator shows the answer for several ranges side by side.

Does a larger sample reduce the curse?

Yes. A larger sample shrinks the standard error, which lowers the bar a result has to clear and raises the power. Significant results from a test with 80% power overstate the true lift by about 1.12× on average; at 50% power it is 1.4×, at 20% power 2.3×, and at 10% power 3.7×.

Is this the same as false positive risk?

No. False positive risk asks whether a significant win is real at all. The winner’s curse asks how large a real win is. A result can be very likely real and still be overstated: a lift that is certainly positive may be half the size the test reported.

Which number should I report?

Report both, and use the shrunk one for forecasts. The observed lift is what the test measured and belongs in the record. The expected true lift is the better guess at what you will see after launch, so it is the one to use when projecting revenue, setting targets, or adding up the impact of several shipped tests.

  1. Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.How far significant results overstate the true effect, and how that depends on power.
  2. van Zwet, E. W., & Cator, E. A. (2021). The Significance Filter, the Winner’s Curse and the Need to Shrink. Statistica Neerlandica, 75(4), 437–452.Why selecting on significance inflates estimates, and the case for shrinkage.
  3. Efron, B. (2011). Tweedie’s Formula and Selection Bias. Journal of the American Statistical Association, 106(496), 1602–1614.Why an estimate that accounts for prior plausibility needs no further correction for selection.
  4. Lee, M. R., & Shen, M. (2018). Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18), 491–499.The winner’s curse in the summed impact of shipped A/B tests.