TESTING HAVEN

III · DIAGNOSE

Peeking Penalty Calculator

What checking early does to your false positive rate

Peeking means checking an A/B test before it is finished and stopping as soon as it looks significant. Every extra look is another chance for noise to cross the line. A test checked 10 times at 95% confidence has a real false positive rate of 19.3%, not 5%.

How the test is checked

Confidence
Test type

Checking a test 10 times and stopping the first time it looks significant gives a real false positive rate of 19.3%, not 5%: 3.9 times what the test claims.

How to peek safely

To keep the overall false positive rate at 5% across 10 looks, call a look significant only when p < 0.0106 (two-tailed), the same threshold at every look. The price is a larger sample for the same power.

If you would rather not fix the number of looks in advance, the sequential testing calculator gives a p-value that stays valid however often you check.

The penalty by number of looks

LooksReal false positive rateTimes the claimed rate
15.0%1.0×
28.3%1.7×
310.7%2.1×
514.2%2.8×
1019.3%3.9×
2024.8%5.0×
5032.0%6.4×
10037.4%7.5×

For a two-tailed test at 95% confidence, with looks spaced evenly through the test and no real difference between the groups.

How the mathematics works

What this calculator does

It answers one question: if you check a test several times and stop the first time it looks significant, how often will a change that does nothing be declared a winner? Enter the number of looks and the confidence level, and it returns the real false positive rate and the threshold you would need at each look to bring that rate back to what you intended.

The size of the problem

For a two-tailed test at 95% confidence, with evenly spaced looks:

  • 5 looks: 14.2%
  • 10 looks: 19.3%
  • 20 looks: 24.8%
  • 100 looks: 37.4%

The rate keeps climbing with more looks. A test that is watched daily for three weeks and stopped at the first significant reading is wrong about a quarter of the time when there is nothing to find.

How it is calculated

With no real difference between the groups, the test statistic follows a random walk as visitors arrive. The chance of that walk crossing the significance boundary at any of a fixed set of looks has no simple formula, but it can be computed exactly: start with the distribution of the statistic at the first look, cut away the part that crossed, carry what remains forward to the next look, and repeat. What is left at the end is the chance of never crossing. The method was published by Armitage, McPherson and Rowe in 1969, and the calculator reproduces their table.

The safe threshold is the same calculation run backwards: the single p-value threshold that, applied at every look, gives exactly the overall rate you wanted. This is known as a Pocock boundary.

What to do instead

  • Decide the sample size first. Use the sample size calculator, run to that number, and read the result once.
  • Plan a few looks and pay for them. Use the stricter threshold above at each look. You will need somewhat more traffic for the same power.
  • Use a method made for watching. The sequential testing calculator gives an always-valid p-value with no limit on the number of looks.

What it assumes

  • The looks are spaced evenly through the test.
  • The test stops at the first look that reaches significance, in either direction for a two-tailed test.
  • There is no real difference. The rates here are false positive rates; with a real difference, early stopping also exaggerates the measured lift (see the winner's curse calculator).
Why does checking a test more often cause false positives?

A p-value wanders as data arrive. With no real difference it stays above 0.05 most of the time, but it dips below now and then. Look once, at the end, and you catch it below 0.05 just 5% of the time. Look repeatedly and stop the first time it dips, and you catch every dip. With 5 looks the chance is 14.2%; with 20 it is 24.8%.

Is it peeking if I look but do not stop?

No. Looking is harmless; acting on what you see is what does the damage. If you decide the sample size in advance and only draw a conclusion at the end, you can watch the dashboard as often as you like. The trouble starts when "it looks significant, let's call it" becomes the stopping rule.

How can I check a test early without the penalty?

Either plan the looks and use a stricter threshold at each one, or use a method built for continuous checking. For 5 planned looks, calling a look significant only when p < 0.0158 keeps the overall rate at 5%; for 10 looks the threshold is p < 0.0106. The sequential testing calculator on this site gives a p-value that stays valid however often you look.

Does it matter when the looks happen?

A little. This calculator assumes looks spaced evenly through the test, which is the standard case and close to a daily or weekly check. Looks bunched together near the end cost less than the same number spread out, because the data barely change between them.

My testing tool shows significance in real time. Is that peeking?

It depends on the statistics behind the display. Some tools use sequential methods designed for continuous monitoring, and their significance is safe to act on. Others show an ordinary fixed-sample p-value recalculated on every refresh, and stopping on that is peeking in its purest form. Check the documentation for words like "sequential", "always-valid" or "group sequential".

  1. Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society: Series A (General), 132(2), 235–244.The false positive rate of repeated significance tests, and the method used to compute it.
  2. Pocock, S. J. (1977). Group Sequential Methods in the Design and Analysis of Clinical Trials. Biometrika, 64(2), 191–199.The constant per-look threshold that restores the intended rate.
  3. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B Tests: Why It Matters, and What to Do About It. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17), 1517–1525.Peeking in online A/B tests, and always-valid alternatives.