TESTING HAVEN

II · ANALYZE

The A/B Test Calculator

Two schools of statistics. One clear verdict.

Your numbers

Control — the original

Variant B — the challenger

Test type
ADVANCED — PRIORS, RISK THRESHOLD & TRAFFIC SPLIT
%

Expected traffic split for the SRM check. Leave blank for an even split.

Enter visitors and conversions for the control and at least one variant — the frequentist and Bayesian analyses run side by side automatically.

For the curious

How the mathematics works

What this calculator does

This A/B test calculator evaluates your experiment two ways at once. You enter visitors and conversions for each group, and it runs a frequentist significance test and a Bayesian analysis side by side: p-values, confidence intervals, and lift on the left; probability to be best, expected loss, credible intervals, and the full posterior probability distribution of each group's conversion rate on the right. The two frameworks answer different questions about the same data, and seeing them together tells you far more than either alone.

It also runs a sample ratio mismatch (SRM) check on your visitor counts automatically — if your traffic split is broken, no amount of statistics can rescue the test, so the calculator warns you before you read anything else.

Every input is encoded in the URL, so you can share a link to your exact result with your team, and export the result card as a PNG for decks and readouts.

The frequentist test

The calculator runs a two-proportion z-test. With n1,n2n_1, n_2 visitors and observed conversion rates p^1,p^2\hat{p}_1, \hat{p}_2 for control and variant, the test statistic is:

z=p^2−p^1p^(1−p^)(1n1+1n2),p^=x1+x2n1+n2z = \frac{\hat{p}_2 - \hat{p}_1}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}, \qquad \hat{p} = \frac{x_1 + x_2}{n_1 + n_2}

where x1,x2x_1, x_2 are the conversion counts and p^\hat{p} is the pooled rate under the null hypothesis that both groups convert identically. The two-tailed p-value is p=2 (1−Φ(∣z∣))p = 2\,(1 - \Phi(|z|)), where Φ\Phi is the standard normal CDF. The confidence interval on the difference uses the unpooled standard error, and the interval on relative lift uses the delta method.

The Bayesian model

The Bayesian panel treats each group's true conversion rate as an uncertain quantity with a Beta prior (uniform Beta(1, 1) by default, configurable under Advanced). Observing xx conversions in nn visitors gives the closed-form posterior

p∣data  ∼  Beta(α0+x,  β0+n−x)p \mid \text{data} \;\sim\; \mathrm{Beta}(\alpha_0 + x,\; \beta_0 + n - x)

which is exactly the probability distribution drawn for each group in the posterior chart. The probability a variant beats control is computed with Evan Miller's exact summation, expected loss uses Chris Stucchio's closed form, and probability-to-be-best across three or more variants plus the credible interval on lift are estimated by Monte Carlo simulation with 100,000 posterior draws.

A worked example

Suppose your control had 10,000 visitors and 1,000 conversions (10.0%), and your variant had 10,000 visitors and 1,120 conversions (11.2%). The pooled rate is 10.6%, giving a standard error of 0.435 percentage points and z=2.75z = 2.75. The two-tailed p-value is 0.006 — well below 0.05 — so the 12% relative lift is statistically significant at the 95% (and even 99%) confidence level. The 95% confidence interval on the relative lift runs from roughly +3.5% to +20.5%: the true effect is very likely positive, but its size is still uncertain.

When to use it

  • After your test has reached its planned sample size (use the sample size calculator before launch).
  • For binary metrics: conversion, sign-up, click-through. For revenue or other continuous metrics, use the revenue-per-visitor calculator.
  • For 2+ variants: add variants and the calculator applies an omnibus chi-square test plus Holm-corrected pairwise comparisons automatically.

Common mistakes

  • Peeking. Checking daily and stopping when p first dips below 0.05 inflates false positives dramatically. Decide the sample size in advance, or use the sequential testing calculator, whose p-values remain valid under continuous monitoring.
  • One-tailed tests chosen after the fact. Halving the p-value by switching to one-tailed after seeing a positive result is p-hacking. Default to two-tailed.
  • Ignoring sample ratio mismatch (SRM). If your 50/50 test shows a badly uneven split of actual traffic, your assignment may be broken and every p-value in the test suspect. This calculator checks for SRM automatically; the standalone SRM checker is still available for checking splits on their own.
  • Multiple comparisons without correction. Each extra variant or metric multiplies your chances of a fluke. Use the built-in Holm correction for A/B/n tests.
  • Confusing significance with size. With enough traffic, a trivial +0.2% lift can be "significant". Read the confidence interval, not just the verdict.
What p-value counts as statistically significant?

By convention, a result is called statistically significant when the p-value is below 0.05, which corresponds to a 95% confidence level. This threshold is a trade-off, not a law of nature: at p < 0.05 you accept a 5% chance of declaring a winner when there is no real difference. For high-stakes or hard-to-reverse changes, many teams require p < 0.01 instead.

Should I use a one-tailed or two-tailed test?

Use a two-tailed test unless you have a specific, pre-registered reason not to. A two-tailed test can detect both improvements and regressions. A one-tailed test reaches significance faster, but it is blind to the possibility that your variant is worse — and switching to one-tailed after seeing the data is a form of p-hacking that inflates your false positive rate.

My test has been running for a week and p is 0.06. Can I keep it running until it hits 0.05?

Only if you planned the longer duration in advance. Repeatedly checking a fixed-horizon test and stopping the moment p dips below 0.05 (called "peeking") badly inflates the false positive rate — with daily checks, a nominal 5% error rate can become 20% or more. Either fix the sample size in advance with a sample size calculator, or use a sequential testing method with always-valid p-values.

Why does this calculator show both frequentist and Bayesian results?

Because they answer different questions about the same data. The frequentist panel answers: "if there were truly no difference, how surprising would this data be?" (p-values and confidence intervals). The Bayesian panel answers: "given this data, what is the probability the variant is better, and how much do I risk by shipping it?" (probability to be best, expected loss, and posterior distributions). Seeing both side by side is the fastest way to understand where the frameworks agree, where they diverge, and why.

What is the SRM check that runs automatically?

SRM stands for sample ratio mismatch. Before you trust any p-value or posterior, the visitor counts you enter are compared against your expected traffic split (even by default, configurable under Advanced) with a chi-square test. If the observed split is far more uneven than random assignment can explain (p < 0.001), the calculator warns you that the test data itself is suspect — a broken split invalidates every metric downstream.

What happens when I test more than one variant (A/B/n)?

Each extra variant adds another comparison, and every comparison carries its own false positive risk. Testing 4 variants at α = 0.05 gives roughly an 18.5% chance of at least one false positive if you do not correct for it. This calculator runs an omnibus chi-square test across all groups and adjusts pairwise p-values with the Holm (default) or Bonferroni correction to keep the overall error rate at your chosen α.

Do visitors and conversions need to be unique counts?

Yes — the test assumes each visitor is an independent trial that either converted or did not. Use unique visitors (or users, or sessions — consistently) and count each converter once. Counting total conversion events, where one user can convert repeatedly, violates the independence assumption and makes the p-value unreliable.