Skip to content
TESTING HAVEN

IV · LEARN

How the Numbers Are Checked

Every statistic here, tested against an independent source

Every statistic on Testing Haven is compared with an answer produced somewhere else: by SciPy or statsmodels, the standard scientific libraries for Python, or by a table printed in a published paper. The table below is rebuilt from the live code each time the site is published. It currently covers 24 checks, 135 reference cases and 403 individual numbers.

Significance of a difference between two conversion ratesTwo-proportion z-testCompared against: statsmodels (proportions_ztest)Reference cases: 6Agrees to 14 significant digits
Significance when conversions are fewFisher’s exact testCompared against: SciPy (fisher_exact)Reference cases: 5Agrees to 11 significant digits
Confidence interval for a conversion rateWilson score intervalCompared against: statsmodels (proportion_confint)Reference cases: 4Identical to 15 digits
Sample ratio mismatch checkChi-square goodness-of-fit testCompared against: SciPy (chisquare)Reference cases: 4Agrees to 12 significant digits
Overall test across several variantsChi-square test of independenceCompared against: SciPy (chi2_contingency)Reference cases: 3Agrees to 13 significant digits
Correction for several comparisonsHolm step-down adjustmentCompared against: statsmodels (multipletests)Reference cases: 3Identical to 15 digits
False discovery rate across many metricsBenjamini–Hochberg adjustmentCompared against: statsmodels (multipletests)Reference cases: 6Identical to 15 digits
Difference in revenue per visitorWelch’s t-test, with its confidence intervalCompared against: SciPy (ttest_ind)Reference cases: 3Agrees to 7 significant digits
Rank-based check for skewed revenueMann–Whitney U testCompared against: SciPy (mannwhitneyu)Reference cases: 3Agrees to 13 significant digits
The beta distribution behind every Bayesian resultBeta cumulative distributionCompared against: SciPy (beta.cdf)Reference cases: 11Agrees to 9 decimal places
Probability the variant beats the controlNumerical integration over both posteriorsCompared against: SciPy numerical integrationReference cases: 9Agrees to 9 decimal places
Expected loss from shipping the variantNumerical integration over both posteriorsCompared against: SciPy numerical integrationReference cases: 9Agrees to 8 significant digits
Probability to be best among several variantsNumerical integration over every posteriorCompared against: SciPy numerical integrationReference cases: 2Agrees to 10 decimal places
Credible interval for the relative liftNumerical integration and root findingCompared against: SciPy numerical integrationReference cases: 3Agrees to 9 decimal places
How much a significant result overstates the truthPower, sign error and exaggeration ratioCompared against: SciPy numerical integrationReference cases: 8Agrees to 11 significant digits
Expected true lift of a winning testShrinkage toward zero with a normal priorCompared against: SciPy numerical integrationReference cases: 6Identical to 15 digits
False positive rate when a test is checked repeatedlyRecursive numerical integrationCompared against: SciPy (joint normal probability)Reference cases: 10Agrees to 5 decimal places
The same, against the original published tableFalse positive rate for 1 to 100 looksCompared against: Armitage, McPherson & Rowe (1969), printed tableReference cases: 9Every printed value reproduced
Chance that a significant result is a false positiveBayes’ rule over ideas that work and ideas that do notCompared against: Kohavi, Deng & Vermeer (2022), printed tableReference cases: 5Every printed value reproduced
Typical effect across a program of testsDerSimonian–Laird random-effects modelCompared against: statsmodels (combine_effects)Reference cases: 4Agrees to 14 significant digits
The revenue-like distribution on the distributions pageTweedie density and chance of zeroCompared against: statsmodels (Tweedie family)Reference cases: 7Agrees to 9 significant digits
Statistical power of a planned testNormal approximationCompared against: Exact power, by adding up every possible outcomeReference cases: 5Within 0.2 percentage points
Sample size for a conversion-rate testNormal approximation with pooled varianceCompared against: statsmodels (NormalIndPower), a different formulaReference cases: 3Within 0.20% of the sample size
Sample size for a revenue testNormal approximation for a difference in meansCompared against: statsmodels (NormalIndPower)Reference cases: 7Within 1 visitor

Reference values were produced with SciPy 1.18.1 and statsmodels 0.15.0. A case is one set of inputs; most cases compare several numbers, such as a test statistic, a p-value and both ends of an interval.

How to read the table

Most rows compare two ways of computing exactly the same quantity, and they agree to many digits. “Agrees to 9 significant digits” means the first nine digits of the site’s answer and the reference answer are the same, in the worst case found. For a p-value of 0.0312345678, any difference would be beyond the last digit shown here.

Three rows are different in kind. Power and the two sample size rows compare the planning formula with a different method: the exact power found by adding up every possible outcome, or a library that uses another standard formula. Planning formulas are approximations, so small differences are expected. The row shows how small: a fraction of a percentage point of power, a fraction of a percent of the sample size, or a single visitor.

One row is limited by its reference. For the peeking calculation, SciPy’s own answer is only accurate to about five decimal places, so that is as far as the comparison can go. The same calculation also reproduces every value in the original 1969 table.

The rule behind it

A number worked out from the formula being tested is not a check. If the formula is wrong, the check is wrong in the same way and passes anyway. So every reference value here comes from separate software, written by other people, or from a table printed in a paper, typed in at the precision the paper printed.

The worked examples in the articles follow the same idea from the other side: the figures in the text are computed by the calculators’ own code when the site is built, so the words cannot drift away from what the calculator shows.

What each calculator uses

A/B test calculator
Two-proportion z-test, switching to Fisher’s exact test when conversions are few. Wilson intervals for each rate. Holm or Bonferroni correction across variants. Bayesian results from beta posteriors by numerical integration, not simulation, so the same inputs always give the same answer.
A/B test report
The same significance test, plus a traffic-split check, the power of the test for the lift you were looking for, and a shrunk estimate of the true lift.
Sample size, power and MDE calculators
The normal approximation for two proportions with pooled variance. With several variants, alpha is divided among the comparisons with the control.
Sequential testing calculator
An always-valid p-value from the mixture sequential probability ratio test, with a normal prior on the effect.
Revenue per visitor calculator
Welch’s t-test with its confidence interval, and the Mann–Whitney U test as a rank-based check.
Sample size for revenue
The normal approximation for a difference in means, from the standard deviation of revenue per visitor.
CUPED calculator
The variance left after adjusting for a pre-test measure: one minus the squared correlation.
False positive risk calculator
Bayes’ rule over ideas that work and ideas that do not, given alpha, power and your success rate.
Winner’s curse calculator
Shrinkage toward zero under a normal prior, and the sign-error and exaggeration calculations of Gelman and Carlin.
Peeking penalty calculator
The chance that any of several evenly spaced looks crosses the line, by recursive numerical integration, and the constant threshold that brings it back to alpha.
Too many metrics calculator
Bonferroni and Holm corrections for the family-wise error rate, and the Benjamini–Hochberg procedure for the false discovery rate.
Program health check
A random-effects model of your past tests to estimate how large real effects usually are, then shrinkage of each result toward that.
SRM checker
A chi-square goodness-of-fit test of the visitor counts.

What the calculators assume

  • Each visitor counts once and independently. If one person can appear many times, or visitors influence each other, the real uncertainty is larger than any calculator will show.
  • The test was read once, at a size fixed in advance. Ordinary p-values and intervals assume that. If you looked early and stopped on a good result, use the peeking penalty or the sequential calculator.
  • The counts are right. A calculator cannot see a tracking fault. The traffic-split check catches one common kind; nothing catches them all.
  • Planning numbers are approximations. Sample sizes and durations are estimates to plan around, not guarantees.

Your numbers stay with you

The calculators run in your browser: nothing you type is sent anywhere to be worked out. Your inputs are kept in the page address, so a result can be bookmarked or shared as a link. The site counts page visits and a few actions, such as “a calculation was completed”, and that counter is given the page name only, never the address with your numbers in it, and it uses no cookies. Opening a shared link works like opening any other link: the address is what your browser asks the web server for.

Kept in order automatically

Every change to the site runs the reference checks above, a browser test of each calculator, and an accessibility scan of every page on a desktop and a phone-sized screen. Once a week the published sources cited across the site are checked to make sure each link still leads to the paper it claims to.

Who publishes it

Testing Haven is published by camploanalytics LLC. The calculators are free and need no account. New terms are explained in the glossary, and the checklist puts the whole method on one page.

  1. Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society: Series A (General), 132(2), 235–244.The printed table the peeking calculation is checked against.
  2. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).The printed table the false positive risk calculation is checked against.
  3. Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.The sign-error and exaggeration calculations.