TESTING HAVEN

II · ANALYZE

Sequential

Peek at results without inflating false positives

Current test data (peek any time)

Enter the current visitors and conversions for both groups. Unlike a fixed-horizon test, you can check this as often as you like.

Why peeking breaks fixed-horizon tests

For the curious

How the mathematics works

What this calculator does

This sequential testing calculator lets you monitor an A/B test continuously and stop the moment the evidence is strong — without the false-positive inflation that makes "peeking" at classical tests infamous. Enter the current visitors and conversions for control and variant, as often as you like, and it returns an always-valid p-value and confidence interval computed with the mixture sequential probability ratio test (mSPRT), the method behind Optimizely's Stats Engine and the "peeking at A/B tests" literature.

Almost every team peeks — the honest fix is not more discipline, it is statistics designed for peeking.

How the mSPRT works

The test tracks a likelihood ratio comparing "no effect" against a mixture of possible effects θ∼N(0,τ2)\theta \sim N(0, \tau^2). For an observed rate difference θ^\hat{\theta} with estimated variance VV, the mixture likelihood ratio has a closed form:

Λ=VV+τ2  exp⁡ ⁣(θ^2 τ22V(V+τ2))\Lambda = \sqrt{\frac{V}{V + \tau^2}}\; \exp\!\left( \frac{\hat{\theta}^2\, \tau^2}{2V(V + \tau^2)} \right)

Under the null hypothesis, Λ\Lambda is a martingale — its expected value never grows, no matter how long you watch it. That is the mathematical reason you may peek freely: the always-valid p-value p=min⁡(1,1/Λ)p = \min(1, 1/\Lambda) can only be pushed below α\alpha by genuine evidence, with probability at most α\alpha over the entire lifetime of the test.

A worked example

Day 6 of a test: control 8,000 visitors / 800 conversions (10.0%), variant 8,000 / 920 (11.5%). A naive fixed-horizon z-test gives p ≈ 0.002 — but you have been checking every day, so that number no longer means what it claims. The mSPRT gives an always-valid p ≈ 0.02. Still below α = 0.05, so you can stop and ship — and this conclusion is honest about the daily monitoring. If the always-valid p had been 0.11, the right move would be to keep collecting, knowing tomorrow's check costs nothing statistically.

When to use it

  • Any test whose results appear on a live dashboard that people actually look at.
  • Tests where shipping early has real value (fixing a losing variant fast).
  • Auditing past "wins" that were declared the day they crossed p < 0.05.
  • When you cannot pre-commit to a sample size — though it is still worth checking the MDE calculator to confirm the test is feasible at all.

Common mistakes

  • Applying sequential thresholds to a fixed-horizon p-value. The p-value on your analytics dashboard is almost certainly fixed-horizon; treating it as safe to monitor is precisely the peeking problem. Use this calculator's p-value instead.
  • Stopping on the naive p-value and "confirming" with mSPRT later. Decide with the always-valid number; the naive one is for curiosity only.
  • Skipping data-quality checks because the stats are fancy. mSPRT assumes valid randomization — run the SRM checker alongside it.
  • Restarting the test after a near-miss. Killing a test at p = 0.06 and relaunching it "fresh" is peeking with extra steps. The martingale only protects you if the test keeps its history. Read more in our peeking guide.
What is an always-valid p-value?

A p-value you can look at whenever you like — after every visitor, every hour, every day — and act on the moment it crosses your threshold, without inflating your false positive rate. Classical p-values are only valid at one pre-planned sample size; always-valid p-values from the mSPRT hold simultaneously at every sample size, which is exactly the guarantee continuous monitoring needs.

Why exactly does peeking break a normal A/B test?

A fixed-horizon p-value is a random number that wobbles as data arrives, and under the null hypothesis it dips below 0.05 by luck about 5% of the time at any single look. If you check 20 times and stop at the first dip, you give yourself 20 chances at that 5% event — the effective false positive rate can climb past 20–30%. The simulation chart on this page shows it happening.

What is the catch compared to a fixed-horizon test?

Sequential tests pay for their flexibility with efficiency: to keep the error guarantee at every possible stopping point, the evidence bar is higher, so a true effect of a given size needs somewhat more data to reach significance than a perfectly-executed fixed-horizon test would need. In practice the trade is worth it — fixed-horizon tests are rarely perfectly executed, and stopping early on strong effects usually recovers the difference.

What is the tau (τ) parameter?

The mSPRT needs a prior guess about the scale of plausible effects, expressed as the standard deviation τ of a normal distribution centered at zero. This calculator defaults τ to 10% of your pooled conversion rate, a reasonable scale for typical A/B lifts. The test stays valid for any τ — a mismatched τ only costs detection speed, never correctness.

Can I use this on a test I already peeked at?

Yes — that is one of its best uses. If you peeked at a fixed-horizon test and it "hit significance", recompute it here. The always-valid p-value tells you whether the evidence is strong enough to survive the peeking you actually did. If mSPRT says significant, you are safe; if not, the fixed-horizon significance was likely a peeking artifact.

When should I still prefer a fixed-horizon test?

When you genuinely will not peek (results are hidden until a scheduled readout), when you need maximum sensitivity per visitor on scarce traffic, or when the runtime is fixed for external reasons anyway (a two-week campaign). In those cases the classical test plus a pre-computed sample size is slightly more efficient. For everything else — dashboards that update live, stakeholders who ask daily — sequential is safer.