TESTING HAVEN

III · DIAGNOSE

Experiment Program Health Check

What your past tests really add up to

A testing program's wins add up to less than they claim, because the tests that reach significance are the lucky ones. Paste your past tests to see how much is real. In the example on this page, 16 tests produce 4 wins claiming +55.9% between them; the lift to expect is +26.2%.

Your past tests

Enter every finished test, including the ones that showed nothing. A list of winners only would make every number below too optimistic.

Confidence

Paste at least 5 past tests, or try the example.

How the mathematics works

What this calculator does

It reads a whole testing program at once. From a list of past tests it estimates how many of your wins are real, what they add up to once the luck is taken out, and how large real lifts typically are in your program. That last figure is the one the winner's curse calculator and the test report ask you to supply, so this page turns a guess into an estimate from your own record.

How it works

Each test ii has an observed lift d^i\hat{d}_i and a standard error sis_i, recovered from its p-value. The true lifts are treated as varying around a program average μ\mu with standard deviation τ\tau. The spread τ\tau is estimated from how much more the observed lifts vary than their standard errors can explain. Each test's expected true lift is then

μ+τ2τ2+si2 (d^i−μ)\mu + \frac{\tau^2}{\tau^2 + s_i^2}\,(\hat{d}_i - \mu)

a noisy test (large sis_i) is pulled most of the way to the average, and a precise one keeps most of its own estimate. The estimate of τ\tau is the standard one from meta-analysis, due to DerSimonian and Laird.

A worked example

The example has 16 tests. 4 reached significance, with observed lifts that compound to +55.9%. Read together, the record suggests most real lifts in this program fall between -6.4% and +10.3%. After shrinkage the same 4 wins are expected to deliver +26.2%, about 47% of the claim, and about 3.9 of them are expected to be real improvements at all.

What it assumes

  • You entered every test. A list of winners only cannot be corrected, because the information about luck is in the tests that were left out.
  • Lifts are relative and the p-values two-tailed. That is what most testing tools report.
  • The wins are independent launches. The totals compound the lifts as if each win were shipped on top of the last and affected the same metric.
  • Real lifts are roughly bell-shaped around an average. A program with a few enormous wins among many duds fits this less well, and its big wins will be shrunk somewhat too much.
What do I need to enter?

One line per finished test, with the observed relative lift and its p-value, and a name if you like. Enter at least 5 tests, and include all of them: the flat ones and the losers as well as the winners. The losers and the flat results are what tell the calculator how much of your winners' lift is luck.

Why is the expected total lower than what my tests reported?

Each observed lift is the true lift plus noise, and a test becomes a win when the two together clear the significance bar. The wins are therefore the tests where the noise ran upward. Add them up and you add up the luck as well. The calculator estimates how much real lifts vary across your program and pulls each observed lift back toward the average by the right amount.

How can a list of tests say how much to shrink each one?

By comparing the spread you see with the spread noise alone would cause. Each test has a known amount of noise, from its sample size. If your observed lifts are no more spread out than that noise predicts, the tests are consistent with all having the same true lift, and each one shrinks a long way. If they are far more spread out, real differences between ideas must be large, and the estimates are trusted more.

How many tests do I need for this to be reliable?

It runs from 5, but the estimate of how much real lifts vary is rough until you have a few dozen. With a short list, treat the output as a direction, not a measurement: the claimed total is too high, and probably by about this much.

Does it matter that my tests were on different pages and metrics?

It matters a little. The method treats your tests as a sample from one program with one typical spread of real lifts. Mixing very different kinds of test, such as pricing changes and button colours, blurs that picture. If you have enough tests, run the check separately for each kind.

  1. DerSimonian, R., & Laird, N. (1986). Meta-analysis in Clinical Trials. Controlled Clinical Trials, 7(3), 177–188.The estimate of how much true effects vary between studies.
  2. Morris, C. N. (1983). Parametric Empirical Bayes Inference: Theory and Applications. Journal of the American Statistical Association, 78(381), 47–55.Empirical Bayes shrinkage, with the uncertainty of the estimated average.
  3. Lee, M. R., & Shen, M. (2018). Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18), 491–499.The winner’s curse in the summed impact of shipped A/B tests.
  4. Efron, B. (2011). Tweedie’s Formula and Selection Bias. Journal of the American Statistical Association, 106(496), 1602–1614.Why shrinkage estimates need no further correction for selection.