The Peeking Problem: Why Checking Results Early Breaks Your Test

Peeking — checking a running test and stopping the moment it looks significant — is the most common way honest teams generate false winners. This guide explains the mechanism, shows the size of the damage, and covers the fix: sequential methods with always-valid p-values.

The mechanism

A p-value is a random quantity that wanders as data accumulates. Under the null hypothesis — no real difference — it dips below 0.05 about 5% of the time at any single, pre-planned look. That guarantee is per-look. If you look every day for a month and stop at the first dip, you take dozens of draws at a 5%-ish event. The chance of at least one false dip compounds: with 10 looks the effective false-positive rate roughly quadruples; with continuous monitoring it can exceed 30%. The interactive simulation on the sequential testing calculator lets you watch an A/A test — two identical experiences — cross into "significance" by luck alone.

Why willpower fails

The peeking incentive is structural, not personal. Results appear on live dashboards. Stakeholders ask "how's the test doing?" in standup. A losing variant bleeds money each day it runs, so early stopping feels responsible. Telling a team "don't look" is a policy that loses to human nature every time. The durable fix is statistics that remain valid when people look — not hoping they won't.

Always-valid p-values

The mixture sequential probability ratio test (mSPRT) tracks a likelihood-ratio process that, under the null, is a nonnegative martingale — a quantity with no tendency to grow no matter how long you watch. The always-valid p-value derived from it can be checked after every visitor; over the whole life of the test, the chance it ever falls below α while the null is true stays at most α. In exchange, it demands somewhat more evidence than a single-look test — the price of unlimited looks.

What to do in practice

For tests on live dashboards, monitor with the sequential calculator and stop when the always-valid p-value crosses your α. For scheduled-readout tests nobody will peek at, pre-commit a sample size with the sample size calculator and read the result once with the significance calculator. And for past "wins" that were stopped the day they crossed p < 0.05: recompute them sequentially. The ones that survive were real; the ones that don't were the peeking.