Testing Haven

Bayesian A/B Test Calculator

Probability to be best, expected loss, and credible intervals

Test data

Control

Variant B

%

Enter your test data and press Calculate to run the Bayesian analysis.

What this calculator does

This Bayesian A/B test calculator treats each variant's true conversion rate as an uncertain quantity and updates that uncertainty with your data. Instead of a p-value, you get answers to the questions stakeholders actually ask: What is the probability the variant is better? How big is the lift likely to be? How much do we risk if we ship it and we are wrong? It reports probability to be best, probability of beating control, expected loss against a configurable risk threshold, a credible interval on relative lift, and a chart of the full posterior distributions.

The model

Conversions are modeled as draws from a Binomial distribution, with a Beta prior on the conversion rate. The Beta is the conjugate prior for the Binomial, so the posterior has a closed form: starting from a prior Beta(α0,β0)\mathrm{Beta}(\alpha_0, \beta_0) and observing xx conversions in nn visitors, the posterior is

p∣data  ∼  Beta(α0+x,  β0+n−x)p \mid \text{data} \;\sim\; \mathrm{Beta}(\alpha_0 + x,\; \beta_0 + n - x)

The probability that variant B beats control A is computed exactly using the closed-form summation

Pr⁡(pB>pA)=∑i=0αB−1B(αA+i,  βA+βB)(βB+i) B(1+i,βB) B(αA,βA)\Pr(p_B > p_A) = \sum_{i=0}^{\alpha_B - 1} \frac{B(\alpha_A + i,\; \beta_A + \beta_B)}{(\beta_B + i)\, B(1 + i, \beta_B)\, B(\alpha_A, \beta_A)}

(Evan Miller's formula), and the expected loss uses Chris Stucchio's closed form. Probability-to-be-best across three or more variants and the credible interval on lift are estimated by Monte Carlo simulation with 100,000 posterior draws.

A worked example

Control: 5,000 visitors, 500 conversions (10.0%). Variant B: 5,000 visitors, 560 conversions (11.2%). With the uniform prior, the posterior for control is Beta(501, 4501) and for B is Beta(561, 4441). The calculator reports roughly a 97.5% probability that B beats control, a median lift of about +12%, and an expected loss of around 0.005% of conversion rate — if B is secretly worse, you stand to lose almost nothing, so shipping is a low-risk decision even though a frequentist test at 99% confidence would still be "inconclusive".

When to use Bayesian over frequentist

  • When you need a decision, not a publication: expected loss gives you a quantified risk to weigh against the cost of keeping the test running.
  • When sample sizes are small and you have solid priors from previous tests on the same funnel.
  • When communicating with non-statisticians: "93% chance B is better" lands better than "p = 0.07, fail to reject the null".
  • Read the deeper comparison in our Bayesian vs frequentist guide.

Common mistakes

  • Treating 95% probability as a magic gate. Combine it with expected loss — a coin-flip probability with negligible downside can still justify shipping.
  • Overconfident priors. An informative prior based on wishful thinking rather than data will drag your posterior toward fiction. Beta(1, 1) is the honest default.
  • Skipping data-quality checks. Bayesian math does not rescue broken randomization — run the SRM checker before believing any posterior.
  • Aggressive optional stopping. Checking constantly and stopping at the first 95% crossing still overstates your win rate; prefer expected-loss decision rules or planned durations.

Frequently Asked Questions

What does "probability to be best" actually mean?

It is the posterior probability that a variant has the highest true conversion rate of all the groups in your test, given the data you observed and your prior. Unlike a p-value — which describes how surprising your data would be if nothing were different — this is a direct statement about what you care about: the chance each variant is the real winner.

What probability is high enough to ship?

There is no universal threshold, but 95% probability of beating control is a common bar, roughly comparable in strictness to p < 0.05. The better practice is to combine it with expected loss: ship when the probability is high AND the expected loss (what you give up if you are wrong) is below a threshold you can tolerate, such as 0.1% of your baseline conversion rate.

What is expected loss and why does it matter?

Expected loss is the average conversion rate you would sacrifice by shipping a variant if it is actually worse than control, weighted by how likely and how large that downside is. It converts uncertainty into a business-relevant risk number. A variant with 90% probability of winning but tiny downside can be a better ship decision than one with 96% probability and a fat tail of bad outcomes.

What prior should I use?

The default Beta(1, 1) prior is uniform — it assumes nothing about your conversion rate and lets the data speak entirely for itself. If you have solid history, you can encode it: a prior of Beta(100, 900) says "I am as confident the rate is around 10% as if I had already seen 1,000 visitors." Informative priors shrink noisy early results toward reality, but a badly chosen prior biases everything, so when in doubt stay uniform.

Can I peek at Bayesian results whenever I want?

Peeking is less catastrophic than with fixed-horizon p-values, but the popular claim that Bayesian tests are "immune to peeking" is overstated. If you repeatedly check probability-to-be-best and stop the moment it crosses 95%, you will still ship more false winners than the number suggests. Decision rules based on expected loss are more robust to optional stopping, and running tests for a pre-planned duration remains good hygiene.

How is the credible interval different from a confidence interval?

A 95% credible interval means: given the data and prior, there is a 95% probability the true lift lies inside this range — the intuitive reading most people mistakenly apply to confidence intervals. A 95% confidence interval instead describes the procedure: 95% of intervals built this way across many experiments would contain the truth.