What this calculator does
It answers one question: if you check a test several times and stop the first time it looks significant, how often will a change that does nothing be declared a winner? Enter the number of looks and the confidence level, and it returns the real false positive rate and the threshold you would need at each look to bring that rate back to what you intended.
The size of the problem
For a two-tailed test at 95% confidence, with evenly spaced looks:
- 5 looks: 14.2%
- 10 looks: 19.3%
- 20 looks: 24.8%
- 100 looks: 37.4%
The rate keeps climbing with more looks. A test that is watched daily for three weeks and stopped at the first significant reading is wrong about a quarter of the time when there is nothing to find.
How it is calculated
With no real difference between the groups, the test statistic follows a random walk as visitors arrive. The chance of that walk crossing the significance boundary at any of a fixed set of looks has no simple formula, but it can be computed exactly: start with the distribution of the statistic at the first look, cut away the part that crossed, carry what remains forward to the next look, and repeat. What is left at the end is the chance of never crossing. The method was published by Armitage, McPherson and Rowe in 1969, and the calculator reproduces their table.
The safe threshold is the same calculation run backwards: the single p-value threshold that, applied at every look, gives exactly the overall rate you wanted. This is known as a Pocock boundary.
What to do instead
- Decide the sample size first. Use the sample size calculator, run to that number, and read the result once.
- Plan a few looks and pay for them. Use the stricter threshold above at each look. You will need somewhat more traffic for the same power.
- Use a method made for watching. The sequential testing calculator gives an always-valid p-value with no limit on the number of looks.
What it assumes
- The looks are spaced evenly through the test.
- The test stops at the first look that reaches significance, in either direction for a two-tailed test.
- There is no real difference. The rates here are false positive rates; with a real difference, early stopping also exaggerates the measured lift (see the winner's curse calculator).