Skip to content
TESTING HAVEN

IV · LEARN

The Peeking Problem: Why Checking Results Early Breaks Your Test

By Testing Haven · Published · Updated

Peeking — checking a running test and stopping the moment it looks significant — is the most common way honest teams generate false winners. This guide explains the mechanism, shows the size of the damage, and covers the fix: sequential methods with always-valid p-values.

The mechanism

A p-value is a random quantity that wanders as data accumulates. Under the null hypothesis — no real difference — it dips below 0.05 about 5% of the time at any single, pre-planned look. That guarantee is per-look. If you look every day for a month and stop at the first dip, you take dozens of draws at a 5%-ish event. The chance of at least one false dip compounds: with 10 looks the effective false-positive rate roughly quadruples; with continuous monitoring it can exceed 30%. The interactive simulation on the sequential testing calculator lets you watch an A/A test — two identical experiences — cross into "significance" by luck alone.

Simulated A/A tests with evenly spaced looks, each look tested at the usual p < 0.05:

Looks at the dataTests that show a false positive
1 (as planned)5%
28%
514%
1019%
2025%
5032%
10037%

The looks are not independent — each one shares most of its data with the last — which is why ten looks give 19% rather than the 40% you would get from ten separate tests. It is still nearly four times the error rate you signed up for.

Looking is not the problem; acting on the look is. If results are hidden until a scheduled readout, or nobody is allowed to stop the test on what they see, the error rate is untouched. The damage starts when a look can end the test.

Why willpower fails

The peeking incentive is structural, not personal. Results appear on live dashboards. Stakeholders ask "how's the test doing?" in standup. A losing variant bleeds money each day it runs, so early stopping feels responsible. Telling a team "don't look" is a policy that loses to human nature every time. The durable fix is statistics that remain valid when people look — not hoping they won't.

Always-valid p-values

The mixture sequential probability ratio test (mSPRT) tracks a likelihood-ratio process that, under the null, is a nonnegative martingale — a quantity with no tendency to grow no matter how long you watch. It starts at 1, and the chance it ever reaches 20 when there is no real effect is at most 1 in 20. The always-valid p-value is simply one over that ratio, so it can be checked after every visitor; over the whole life of the test, the chance it ever falls below α while the null is true stays at most α. In exchange, it demands somewhat more evidence than a single-look test — the price of unlimited looks.

The guarantee holds up in simulation. In A/A tests checked 200 times each, an ordinary z-test crossed p < 0.05 at some point in 42% of runs. The always-valid p-value did so in about 2%.

The method has one tuning choice: the scale of effects you consider plausible, written τ. Our calculator sets it to 10% of the pooled conversion rate, a sensible scale for typical lifts. The guarantee holds for any τ fixed in advance; a poor choice costs detection speed, not validity.

Assume the p-value on your analytics dashboard is not always-valid unless the vendor's documentation says it is sequential. Most are ordinary fixed-horizon p-values recomputed on every refresh, which is the peeking problem with a nicer chart.

A worked example

Day 3. Control: 4,000 visitors, 400 conversions (10.0%). Variant: 4,000 visitors, 460 conversions (11.5%). The ordinary z-test says p = 0.030 — "significant", and the temptation to call it is strong. The always-valid p-value is 0.35. Nothing has been established yet; keep going.

Day 6. Same rates, twice the data: 8,000 visitors and 800 conversions against 8,000 and 920. The ordinary p-value is now 0.002. The always-valid p-value is 0.0496, just under 0.05, and its confidence interval on the difference runs from barely above zero to +3.0 percentage points. You may stop here, and the conclusion is honest about the fact that you have been looking every day. At this sample size the sequential test is asking for roughly z > 3.06 where a single planned look would accept 1.96.

Day 8, if you had kept going. 10,000 visitors and 1,000 conversions against 10,000 and 1,150: the always-valid p-value falls to 0.017. Evidence that is real keeps accumulating; evidence that was luck drifts back.

The price of unlimited looks

Sequential tests pay for flexibility with sample size, but less than you might fear. We simulated a true 10% relative lift on a 10% baseline (10% against 11%). A fixed-horizon test needs 14,751 visitors per arm for 80% power. The mSPRT, monitored throughout, had already stopped in a quarter of runs by 8,500 per arm and in about half by the fixed-horizon sample size — but it took about 25,000 per arm before 80% of runs had stopped. Clear effects finish early; marginal ones take longer. Check with the MDE & duration calculator that the test is feasible at all before you start.

Other ways to look early

If your looks are scheduled — a weekly readout, say — a group-sequential design is the classical alternative: plan the number of looks in advance and test each at a stricter threshold. With five planned looks, Pocock's rule tests each at about p < 0.016, which keeps the overall false-positive rate at 5%. The mSPRT suits the opposite situation, where nobody can say in advance how often the dashboard will be opened. Bayesian monitoring is not a third option: stopping on a probability threshold inflates errors in the same way (see Bayesian vs frequentist).

What to do in practice

For tests on live dashboards, monitor with the sequential calculator and stop when the always-valid p-value crosses your α. For scheduled-readout tests nobody will peek at, pre-commit a sample size with the sample size calculator and read the result once with the significance calculator. And for past "wins" that were stopped the day they crossed p < 0.05: recompute them sequentially. The ones that survive were real; the ones that don't were the peeking.

Peeking in disguise

  • "One more week." Extending a test because it is close to significance is a second look with a decision attached.
  • Restarting after a near-miss. Killing a test at p = 0.06 and relaunching it fresh throws away the history that the error guarantee depends on.
  • Stopping losers early on the ordinary p-value. Peeking in one direction is still peeking.
  • Watching many segments. Daily looks multiplied by ten segments is a great many chances for a fluke.
  • Skipping data-quality checks. Sophisticated statistics still assume valid randomization — run the SRM checker alongside.
  1. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B Tests: Why It Matters, and What to Do About It. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17), 1517–1525.The peeking problem and always-valid p-values.
  2. Johari, R., Pekelis, L., & Walsh, D. (2015). Always Valid Inference: Bringing Sequential Analysis to A/B Testing. arXiv:1512.04922.The theory of always-valid inference.
  3. Pocock, S. J. (1977). Group Sequential Methods in the Design and Analysis of Clinical Trials. Biometrika, 64(2), 191–199.Group sequential testing with a fixed number of planned looks.
  4. Robinson, D. (2015). Is Bayesian A/B Testing Immune to Peeking? Not Exactly. Variance Explained.Why Bayesian tests are not immune to peeking.