Skip to content
TESTING HAVEN

II · ANALYZE

Bayesian A/B Test Calculator

Probability to be best, expected loss, and credible intervals

A Bayesian A/B test gives the probability that a variant really beats control, and what you stand to lose if you ship it and are wrong. For example, 5,000 visitors per group converting at 10.0% and 11.2% gives a 97.4% probability that the variant is better.

Control — the original

Variant B — the challenger

Test type
ADVANCED — PRIORS, RISK THRESHOLD & TRAFFIC SPLIT
%

Expected traffic split for the SRM check. Leave blank for an even split.

Enter visitors and conversions for the control and at least one variant — the frequentist and Bayesian analyses run side by side automatically.

How the mathematics works

What this calculator does

This Bayesian A/B test calculator treats each variant's true conversion rate as an uncertain quantity and updates that uncertainty with your data. Instead of a p-value, you get answers to the questions stakeholders actually ask: What is the probability the variant is better? How big is the lift likely to be? How much do we risk if we ship it and we are wrong? It reports probability to be best, probability of beating control, expected loss against a configurable risk threshold, a credible interval on relative lift, and a chart of the full posterior distributions.

The same tool also runs the frequentist analysis of your data side by side — p-values, confidence intervals, and lift — plus an automatic sample ratio mismatch check. Comparing the two frameworks on identical data is the fastest way to understand what each one is (and is not) telling you.

The model

Conversions are modeled as draws from a Binomial distribution, with a Beta prior on the conversion rate. The Beta is the conjugate prior for the Binomial, so the posterior has a closed form: starting from a prior Beta(α0,β0)\mathrm{Beta}(\alpha_0, \beta_0) and observing xx conversions in nn visitors, the posterior is

p∣data  ∼  Beta(α0+x,  β0+n−x)p \mid \text{data} \;\sim\; \mathrm{Beta}(\alpha_0 + x,\; \beta_0 + n - x)

The probability that variant B beats control A, and the expected loss of shipping B, are integrals over the two posteriors:

Pr⁡(pB>pA)=∫01FA(x) fB(x) dx,E[max⁡(pA−pB, 0)]=∫01FB(x) (1−FA(x)) dx\Pr(p_B > p_A) = \int_0^1 F_A(x)\, f_B(x)\, dx, \qquad \mathbb{E}\bigl[\max(p_A - p_B,\, 0)\bigr] = \int_0^1 F_B(x)\,\bigl(1 - F_A(x)\bigr)\, dx

where FF and ff are each posterior's CDF and density. The calculator evaluates them by numerical integration, and does the same for probability-to-be-best across several variants and for the credible interval on lift. Nothing is simulated, so results are exact to many decimal places, identical every time you load the page, and unaffected by non-integer priors or very large samples.

A worked example

Control: 5,000 visitors, 500 conversions (10.0%). Variant B: 5,000 visitors, 560 conversions (11.2%). With the uniform prior, the posterior for control is Beta(501, 4501) and for B is Beta(561, 4441). The calculator reports a 97.4% probability that B beats control, a median lift of +12.0%, and an expected loss of 0.006% of conversion rate — 0.06% of the baseline. If B is secretly worse, you stand to lose almost nothing.

The frequentist reading of the same data is p = 0.051, just short of significance at 95%, and the Bayesian probability is likewise just short of the 97.5% bar that matches a two-tailed test. The two frameworks agree that the evidence is promising but not yet decisive; what the Bayesian reading adds is the expected loss, which tells you how little is at stake if you ship anyway.

When to use Bayesian over frequentist

  • When you need a decision, not a publication: expected loss gives you a quantified risk to weigh against the cost of keeping the test running.
  • When sample sizes are small and you have solid priors from previous tests on the same funnel.
  • When communicating with non-statisticians: "93% chance B is better" lands better than "p = 0.14, fail to reject the null".
  • Read the deeper comparison in our Bayesian vs frequentist guide.

Common mistakes

  • Treating 95% probability as a magic gate. Combine it with expected loss — a coin-flip probability with negligible downside can still justify shipping.
  • Overconfident priors. An informative prior based on wishful thinking rather than data will drag your posterior toward fiction. Beta(1, 1) is the honest default.
  • Skipping data-quality checks. Bayesian math does not rescue broken randomization — this calculator runs a sample ratio mismatch check on your visitor counts automatically, and the standalone SRM checker remains available for checking splits on their own.
  • Aggressive optional stopping. Checking constantly and stopping at the first 95% crossing still overstates your win rate; prefer expected-loss decision rules or planned durations.
What does "probability to be best" actually mean?

It is the posterior probability that a variant has the highest true conversion rate of all the groups in your test, given the data you observed and your prior. Unlike a p-value — which describes how surprising your data would be if nothing were different — this is a direct statement about what you care about: the chance each variant is the real winner.

What probability is high enough to ship?

There is no universal threshold. A 95% probability of beating control is a common bar, but note what it corresponds to: with a flat prior it is about as strict as a one-tailed test at p < 0.05. The equivalent of the usual two-tailed p < 0.05 is 97.5%, which is the bar this calculator applies when your test is two-tailed. The better practice is to combine it with expected loss: ship when the probability is high AND the expected loss (what you give up if you are wrong) is below a threshold you can tolerate, such as 0.1% of your baseline conversion rate.

What is expected loss and why does it matter?

Expected loss is the average conversion rate you would sacrifice by shipping a variant if it is actually worse than control, weighted by how likely and how large that downside is. It converts uncertainty into a business-relevant risk number. A variant with 90% probability of winning but tiny downside can be a better ship decision than one with 96% probability and a fat tail of bad outcomes.

What prior should I use?

The default Beta(1, 1) prior is uniform — it assumes nothing about your conversion rate and lets the data speak entirely for itself. If you have solid history, you can encode it: a prior of Beta(100, 900) says "I am as confident the rate is around 10% as if I had already seen 1,000 visitors." Informative priors shrink noisy early results toward reality, but a badly chosen prior biases everything, so when in doubt stay uniform.

Can I peek at Bayesian results whenever I want?

Peeking is less catastrophic than with fixed-horizon p-values, but the popular claim that Bayesian tests are "immune to peeking" is overstated. If you repeatedly check probability-to-be-best and stop the moment it crosses 95%, you will still ship more false winners than the number suggests. Decision rules based on expected loss are more robust to optional stopping, and running tests for a pre-planned duration remains good hygiene.

How is the credible interval different from a confidence interval?

A 95% credible interval means: given the data and prior, there is a 95% probability the true lift lies inside this range — the intuitive reading most people mistakenly apply to confidence intervals. A 95% confidence interval instead describes the procedure: 95% of intervals built this way across many experiments would contain the truth.

  1. Stucchio, C. (2015). Bayesian A/B Testing at VWO. Visual Website Optimizer technical whitepaper.Expected loss as a Bayesian decision rule.
  2. Miller, E. (2015). Formulas for Bayesian A/B Testing. evanmiller.org.Closed-form formulas the calculator’s integration is tested against.
  3. Casella, G., & Berger, R. L. (1987). Reconciling Bayesian and Frequentist Evidence in the One-Sided Testing Problem. Journal of the American Statistical Association, 82(397), 106–111.Why a one-sided p-value and a Bayesian probability line up.
  4. Robinson, D. (2015). Is Bayesian A/B Testing Immune to Peeking? Not Exactly. Variance Explained.Why stopping at the first high probability still inflates errors.

Every calculation here is tested against an independent source. How the numbers are checked